ranqiqiang commented on a change in pull request #16478: URL: https://github.com/apache/flink/pull/16478#discussion_r669545512
########## File path: docs/content.zh/docs/concepts/stateful-stream-processing.md ########## @@ -24,342 +24,227 @@ under the License. # 有状态流处理 -## What is State? +## 什么是状态? -While many operations in a dataflow simply look at one individual *event at a -time* (for example an event parser), some operations remember information -across multiple events (for example window operators). These operations are -called **stateful**. +虽然数据流中的很多操作一次只着眼于一个单独的事件(例如事件解析器),但有些操作会记住多个事件的信息(例如窗口算子)。 +这些操作称为**有状态的**(stateful)。 -Some examples of stateful operations: +有状态操作的一些示例: - - When an application searches for certain event patterns, the state will - store the sequence of events encountered so far. - - When aggregating events per minute/hour/day, the state holds the pending - aggregates. - - When training a machine learning model over a stream of data points, the - state holds the current version of the model parameters. - - When historic data needs to be managed, the state allows efficient access - to events that occurred in the past. + - 当应用程序搜索某些事件模式时,状态将存储到目前为止遇到的事件序列。 + - 当每分钟/每小时/每天聚合事件时,状态会持有待处理的聚合。 + - 当在数据点的流上训练一个机器学习模型时,状态会保存模型参数的当前版本。 + - 当需要管理历史数据时,状态允许有效访问过去发生的事件。 -Flink needs to be aware of the state in order to make it fault tolerant using +Flink 需要知道状态以便使用 [checkpoints]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) -and [savepoints]({{< ref "docs/ops/state/savepoints" >}}). +和 [savepoints]({{< ref "docs/ops/state/savepoints" >}}) 进行容错。 -Knowledge about the state also allows for rescaling Flink applications, meaning -that Flink takes care of redistributing state across parallel instances. +关于状态的知识也允许我们重新调节 Flink 应用程序,这意味着 Flink 负责跨并行实例重新分布状态。 -[Queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) allows you to access state from outside of Flink during runtime. +[可查询的状态]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})允许你在运行时从 Flink 外部访问状态。 -When working with state, it might also be useful to read about [Flink's state -backends]({{< ref "docs/ops/state/state_backends" >}}). Flink -provides different state backends that specify how and where state is stored. +在使用状态时,阅读 [Flink 的状态后端]({{< ref "docs/ops/state/state_backends" >}})可能也很有用。 +Flink 提供了不同的状态后端,用于指定状态存储的方式和位置。 {{< top >}} -## Keyed State - -Keyed state is maintained in what can be thought of as an embedded key/value -store. The state is partitioned and distributed strictly together with the -streams that are read by the stateful operators. Hence, access to the key/value -state is only possible on *keyed streams*, i.e. after a keyed/partitioned data -exchange, and is restricted to the values associated with the current event's -key. Aligning the keys of streams and state makes sure that all state updates -are local operations, guaranteeing consistency without transaction overhead. -This alignment also allows Flink to redistribute the state and adjust the -stream partitioning transparently. - -{{< img src="/fig/state_partitioning.svg" alt="State and Partitioning" class="offset" width="50%" >}} - -Keyed State is further organized into so-called *Key Groups*. Key Groups are -the atomic unit by which Flink can redistribute Keyed State; there are exactly -as many Key Groups as the defined maximum parallelism. During execution each -parallel instance of a keyed operator works with the keys for one or more Key -Groups. - -## State Persistence - -Flink implements fault tolerance using a combination of **stream replay** and -**checkpointing**. A checkpoint marks a specific point in each of the -input streams along with the corresponding state for each of the operators. A -streaming dataflow can be resumed from a checkpoint while maintaining -consistency *(exactly-once processing semantics)* by restoring the state of the -operators and replaying the records from the point of the checkpoint. - -The checkpoint interval is a means of trading off the overhead of fault -tolerance during execution with the recovery time (the number of records that -need to be replayed). - -The fault tolerance mechanism continuously draws snapshots of the distributed -streaming data flow. For streaming applications with small state, these -snapshots are very light-weight and can be drawn frequently without much impact -on performance. The state of the streaming applications is stored at a -configurable place, usually in a distributed file system. - -In case of a program failure (due to machine-, network-, or software failure), -Flink stops the distributed streaming dataflow. The system then restarts the -operators and resets them to the latest successful checkpoint. The input -streams are reset to the point of the state snapshot. Any records that are -processed as part of the restarted parallel dataflow are guaranteed to not have -affected the previously checkpointed state. +## 键控状态(Keyed State) + +键控状态被维护在一个可以认为是键/值存储的地方。状态和有状态算子读取的流一起被严格地分区和分布。 +因此,只能在*键控流*(keyed streams)上访问键/值状态,即在键控的/分区的数据交换后, +并且仅限于和当前事件的键相关联的值。对齐流和状态的键确保所有状态更新都是本地操作,保证一致性而没有事务开销。 +这个对齐也允许 Flink 重新分布状态并透明地调整流的分区。 + +{{< img src="/fig/state_partitioning.svg" alt="状态和分区" class="offset" width="50%" >}} + +键控状态被进一步组织成*键组*(Key Groups)。键组是 Flink 可以重新分布键控状态的原子单元;有与定义的最大并行度一样多的键组。 +在执行期间,每个键控算子的并行实例和一个或多个键组的键一起使用。 + +## 状态持久化 + +Flink 使用**流重放**(stream replay)和**检查点**(checkpointing)的组合来实现容错。 +检查点标记了每一个输入流的特定点和每个算子的相应状态。 流式数据流可以从检查点恢复, +同时通过重新恢复算子状态并从检查点重放记录来保证一致性 *(精确一次处理语义)*。 + +检查点间隔是一种权衡执行期间容错开销与恢复时间(需要重放的记录数)的方法。 + +容错机制不断地绘制分布式数据流快照。对于小状态的流式应用,它们的快照非常轻量,可以被被频繁绘制,对性能没有太大影响。 +流式应用的状态存储在一个可配置的地方,通常是在分布式文件系统中。 + +如果程序出现故障(由于机器、网络或者软件故障),Flink 会停止分布式数据流。之后系统会重启算子并且将它们重置到最近的成功检查点。 +输入流被重置到状态快照的点。作为重新启动的并行数据流的一部分被处理的任何记录都保证不会影响之前检查点的状态。 {{< hint warning >}} -By default, checkpointing is disabled. See [Checkpointing]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) for details on how to enable and configure checkpointing. +默认情况下,检查点是被禁用的。关于如何启用和配置检查点的详细信息,请参阅[检查点]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}})。 {{< /hint >}} {{< hint info >}} -For this mechanism to realize its full guarantees, the data -stream source (such as message queue or broker) needs to be able to rewind the -stream to a defined recent point. [Apache Kafka](http://kafka.apache.org) has -this ability and Flink's connector to Kafka exploits this. See [Fault -Tolerance Guarantees of Data Sources and Sinks]({{< ref "docs/connectors/datastream/guarantees" >}}) for more information about the guarantees -provided by Flink's connectors. +为了让该机制实现其完全保证,数据流的源(例如消息队列或 broker)需要能够将流回滚到定义的最近点。 +[Apache Kafka](http://kafka.apache.org) 具有这个能力,Flink 的 Kafka +连接器利用了这一点。关于 Flink 连接器提供的保证的更多信息请参阅[数据源和 sink 的容错保证]({{< ref "docs/connectors/datastream/guarantees" >}}) {{< /hint >}} {{< hint info >}} -Because Flink's checkpoints are realized through distributed -snapshots, we use the words *snapshot* and *checkpoint* interchangeably. Often -we also use the term *snapshot* to mean either *checkpoint* or *savepoint*. +由于 Flink 的检查点是通过分布式快照实现的,所以我们交替使用*快照*(snapshot)和*检查点*(checkpoint)两个词。 +通常我们也使用术语*快照*来表示*检查点*或*保存点*(savepoint)。 {{< /hint >}} -### Checkpointing - -The central part of Flink's fault tolerance mechanism is drawing consistent -snapshots of the distributed data stream and operator state. These snapshots -act as consistent checkpoints to which the system can fall back in case of a -failure. Flink's mechanism for drawing these snapshots is described in -"[Lightweight Asynchronous Snapshots for Distributed -Dataflows](http://arxiv.org/abs/1506.08603)". It is inspired by the standard -[Chandy-Lamport algorithm](http://research.microsoft.com/en-us/um/people/lamport/pubs/chandy.pdf) -for distributed snapshots and is specifically tailored to Flink's execution -model. - -Keep in mind that everything to do with checkpointing can be done -asynchronously. The checkpoint barriers don't travel in lock step and -operations can asynchronously snapshot their state. - -Since Flink 1.11, checkpoints can be taken with or without alignment. In this -section, we describe aligned checkpoints first. - -#### Barriers - -A core element in Flink's distributed snapshotting are the *stream barriers*. -These barriers are injected into the data stream and flow with the records as -part of the data stream. Barriers never overtake records, they flow strictly in -line. A barrier separates the records in the data stream into the set of -records that goes into the current snapshot, and the records that go into the -next snapshot. Each barrier carries the ID of the snapshot whose records it -pushed in front of it. Barriers do not interrupt the flow of the stream and are -hence very lightweight. Multiple barriers from different snapshots can be in -the stream at the same time, which means that various snapshots may happen -concurrently. +### 检查点 + +Flink 容错机制的核心部分是绘制分布式数据流和算子状态的一致快照。 这些快照充当一致性检查点, +系统可以在发生故障时可以回退到这些检查点。[分布式数据流的轻量级异步快照](http://arxiv.org/abs/1506.08603)中描述了 Flink 绘制这些快照的机制。 +它受到标准 [Chandy-Lamport 算法](http://research.microsoft.com/en-us/um/people/lamport/pubs/chandy.pdf)的启发, +专门针对 Flink 的执行模型量身定制。 + +请记住,与检查点有关的一切都可以异步完成。检查点屏障(checkpoint barrier)不会在锁定步骤中移动,操作可以异步快照它们的状态。 Review comment: "不会在锁定步骤中移动" 这个感觉也有点生硬 是要描述: checkpoint barrier 不会(加锁)阻塞数据流,所有操作可以异步快照他们的状态吗? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org