[
https://issues.apache.org/jira/browse/FLINK-40932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40932:
-----------------------------------
Labels: pull-request-available (was: )
> Arm state schema evolution for ListState and MapState of RowData
> ----------------------------------------------------------------
>
> Key: FLINK-40932
> URL: https://issues.apache.org/jira/browse/FLINK-40932
> Project: Flink
> Issue Type: Sub-task
> Components: API / Type Serialization System, Runtime / State Backends
> Reporter: Weiqing Yang
> Assignee: Weiqing Yang
> Priority: Major
> Labels: pull-request-available
>
> FLIP-527 covers list-element and map-value migration as well as value state.
> From the FLIP:
> > list state (whose elements are stored individually) and map state (whose
> > values are stored per entry) are migrated by applying the same migrate
> > method to each element or value as it is read — there is no separate
> > element hook.
> FLINK-40298 arms only a state's own value serializer, so `ListState<RowData>`
> and `MapState<K, RowData>` currently fail closed: an evolving schema is
> rejected on restore rather than migrated. This sub-task extends the arming
> seam one structural level to cover them.
> The runtime already descends to exactly these two serializers during
> migration (`RocksDBListState#migrateSerializedValue` unwraps the element
> serializer, `RocksDBMapState#migrateSerializedValue` unwraps the map value
> serializer), so this is an arming change rather than a new migration path.
> Scope:
> - Extend the arming helper with a `ListSerializer` branch (arm the element
> serializer) and a `MapSerializer` branch (arm the value serializer, never the
> key serializer).
> - The descent is exactly one structural level. A `RowData` serializer two or
> more levels down, as in the interval-join shape `MapState<Long,
> List<Tuple2<RowData, Boolean>>>`, must remain unarmed and continue to fail
> closed.
> - Map keys are never armed; key-schema changes remain incompatible.
> - Fail closed at the consuming end. Extending the descent means an armed
> serializer can now reach operator state and broadcast state, because a
> `StateDescriptor` caches its serializer per instance and that cache is shared
> between the keyed and non-keyed stores. `DefaultOperatorStateBackend` accepts
> `compatibleAfterMigration` and never migrates, so it must reject an armed
> serializer instead. This adds a small change in `flink-runtime` alongside the
> `flink-core` one.
> - Tests covering list-element and map-value migration, a test asserting the
> two-level shape is still rejected, and tests for the operator-state and
> broadcast-state rejection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
