[
https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534310#comment-17534310
]
Roman Khachatryan commented on FLINK-23143:
-------------------------------------------
> The added methods we implement just mock what we do when restore (from
> rocksdb or hashmap).
Yes, but we call them at most once on restore currently.
> They will judge before adding new state. And judging will also works when
> users access.
In the PR (prototype), there is no such judgement, right? Do you mean it should
be added?
Does that mean postponing state migration until user access?
I think ideally, state should be migrated before modifying it (i.e. on reading
metadata records from changelog); otherwise, there might be data loss or
exception when serializing state changes in RocksDB. WDYT?
Besides that, what about updating TTL? If we return existing state, than TTL
settings won't be updated, right?
> Maybe we could introduce a new more universal class to wrap these
> information, WDYT?
In my opinion, the problem with RegisteredKeyValueStateBackendMetaInfo is not
its type; but rather the fact that it should be created by the (nested) state
backend, instead of being injected from the outside.
ChangelogBackend metaInfo and e.g. RocksDBBackend metaInfo don't have to be the
same; and the former shouldn't know how to create metaInfo for the latter.
It looks like the [design document and considering to expose upgrade
method|https://issues.apache.org/jira/browse/FLINK-23143?focusedCommentId=17529581&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17529581]
would be helpful after all.
> Support state migration
> -----------------------
>
> Key: FLINK-23143
> URL: https://issues.apache.org/jira/browse/FLINK-23143
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during
> recovery; on 1st user access, it doesn't update metadata nor migrate state
> (as opposed to other backends).
>
> The proposed solution is to
> # wrap serializers (and maybe other objects) in getOrCreateKeyedState
> # store wrapping objects in a new map keyed by state name
> # pass wrapped objects to delegatedBackend.createInternalState
> # on 1st user access, lookup wrapper and upgrade its wrapped serializer
> This should be done for both KV/PQ states.
>
> See also [https://github.com/apache/flink/pull/15420#discussion_r656934791]
>
> cc: [~yunta]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
