sepuri sai krishna created FLINK-40796:
------------------------------------------
Summary: State processor API no longer rejects key types that
cannot be hashed reliably
Key: FLINK-40796
URL: https://issues.apache.org/jira/browse/FLINK-40796
Project: Flink
Issue Type: Bug
Components: API / State Processor
Affects Versions: 2.4.0
Reporter: sepuri sai krishna
FLINK-40751 replaced `stream.keyBy(keySelector)` in
`StateBootstrapTransformation` with a `PartitionTransformation` built directly,
so the key type validation that `KeyedStream` performs is no longer reached.
`OneInputStateTransformation#keyBy` only extracts the key type, so nothing
validates it now.
Before, bootstrapping keyed state with an array key failed at build time:
{code}
InvalidProgramException: Type byte[] cannot be used as key. Contained
UNSUPPORTED key types: byte[]. Look at the keyBy() documentation for the
conditions a type has to satisfy in order to be eligible for a key.
{code}
On master the same job builds, runs and writes a savepoint. The same applies to
an enum key and to a POJO that does not override `hashCode`.
The check comes from FLINK-5874, which restricted key types in the DataStream
API because such a key cannot be hashed reliably. Without it the state is not
partitioned consistently: two equal `byte[] {1, 2, 3}` keys can be assigned to
different key groups (48 and 39 in a local run with `maxParallelism` 128). A
job restoring the savepoint would reject the key type in its own `keyBy`, so
the effect is that an up-front error is replaced by a savepoint that cannot be
used.
Reported by Gyula Fora on https://github.com/apache/flink/pull/29268 as worth
fixing with a test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
