[
https://issues.apache.org/jira/browse/FLINK-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-23344:
--------------------------------------
Description:
Inject TaskStateRegistry added in FLINK-23139 into existing state backends and
use it there to manage private state.
*Should only take effect if changelog.enabled = true.*
*Related task*: prevent RocksDB from re-uploading state if checkpoints weren't
confirmed (or were aborted). Please implement or create a seperate ticket.
An alternative solution that was considered is to analyze the returned snapshot
for state to manage.
[https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.9dxopqajsy7]
was:
(materialized and non-materialized, only changelog backend)
TM should own changelog backend state to prevent re-uploading state on
checkpoint abortion (or missing confirmation). A simpler solution to only own
aborted state is less maintanable in the long run.
For that, on TM state should be tracked and discarded (on
subsumption+materialization; on shutdown). Both materialized and
non-materialized.
See [state ownership design
doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing],
in particular [Tracking private
state|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.osle3oyzw0bx].
Rescaling not supported until shared state management implemented.
Externalized checkpoints and savepoints should be supported (or please create a
separate ticket).
Retained checkpoints is a separated ticket: FLINK-23251
> State ownership: track and discard private state (rocks, heap)
> --------------------------------------------------------------
>
> Key: FLINK-23344
> URL: https://issues.apache.org/jira/browse/FLINK-23344
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Priority: Major
> Fix For: 1.14.0
>
>
> Inject TaskStateRegistry added in FLINK-23139 into existing state backends
> and use it there to manage private state.
>
> *Should only take effect if changelog.enabled = true.*
>
> *Related task*: prevent RocksDB from re-uploading state if checkpoints
> weren't confirmed (or were aborted). Please implement or create a seperate
> ticket.
>
> An alternative solution that was considered is to analyze the returned
> snapshot for state to manage.
>
> [https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.9dxopqajsy7]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
