rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791467158
##########
File path: docs/content/docs/ops/state/state_backends.md
##########
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
+
+With Changelog enabled, Flink uploads state changes continuously, forming a
changelog. On checkpoint, only the relevant
+part of this changelog needs to be uploaded. Independently, configured state
backend is checkpointed in the
+background periodically. Upon successful upload, changelog is truncated.
+
+As a result, asynchronous phase is reduced, as well as synchronous phase (in
particular, long-tail).
+
+On the flip side, resource usage is higher:
+
+- more files are created on DFS
+- more IO bandwidth is used to upload
+- more CPU used to serialize state changes
+- more memory used by Task Managers to buffer state changes
+- todo: more details after testing, maybe link to blogpost
+
+Recovery time is another thing to consider. Depending on the
`state.backend.changelog.periodic-materialize.interval`,
+changelog can become lengthy and replaying it may take more time. However,
recovery time combined with checkpoint
+duration will likely be still lower than in non-changelog setup, providing
lower end-to-end latency even in failover
+case.
Review comment:
> I also saw a lot of cases that recovery time increased by tens of
seconds but checkpoint duration does not decrease that much.
If checkpoint duration doesn't decrease significantly then it's probably not
a suitable case to use Changelog. So the question I think is what's the ratio
between these times and how to put it in the docs.
> I do not think we should mix recovery time with checkpoint duration here.
Reducing **effective** recovery time is actually one of the goals of the
project (please see FLIP
[Motivation](https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints#FLIP158:Generalizedincrementalcheckpoints-Motivation)
and discussions); and the mean to achieve this is by having less data to
replay for the whole pipeline (while having data to replay by Changelog on Task
level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
