[
https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539537#comment-17539537
]
Hangxiang Yu commented on FLINK-23143:
--------------------------------------
I have updated the pr.
> Does that mean postponing state migration until user access?
I think ideally, state should be migrated before modifying it (i.e. on reading
metadata records from changelog); otherwise, there might be data loss or
exception when serializing state changes in RocksDB. WDYT?
Currently, state migration may happen not only when user access, but also
before modifing it as you said (considering materilization part may not be
included in snapshot).
> Besides that, what about updating TTL? If we return existing state, than TTL
> settings won't be updated, right?
Currently, ChangelogStateFactory will be disposed while finishing restore. So
there is no state cache for ChangelogKeyedStateBackend and all TTL of states
will be updated.
> ChangelogBackend metaInfo and e.g. RocksDBBackend metaInfo don't have to be
> the same; and the former shouldn't know how to create metaInfo for the latter.
Sure, I agree. So I have change the methods (createInternalState and create) to
add extra paramater. But I am not sure whether there is a better solution about
interface change.
> Support state migration
> -----------------------
>
> Key: FLINK-23143
> URL: https://issues.apache.org/jira/browse/FLINK-23143
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during
> recovery; on 1st user access, it doesn't update metadata nor migrate state
> (as opposed to other backends).
>
> The proposed solution is to
> # wrap serializers (and maybe other objects) in getOrCreateKeyedState
> # store wrapping objects in a new map keyed by state name
> # pass wrapped objects to delegatedBackend.createInternalState
> # on 1st user access, lookup wrapper and upgrade its wrapped serializer
> This should be done for both KV/PQ states.
>
> See also [https://github.com/apache/flink/pull/15420#discussion_r656934791]
>
> cc: [~yunta]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
