sjwiesman commented on a change in pull request #13912: URL: https://github.com/apache/flink/pull/13912#discussion_r566412417
########## File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/storage/JobManagerCheckpointStorage.java ########## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.state.storage; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.api.common.JobID; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.core.fs.Path; +import org.apache.flink.runtime.state.CheckpointStorage; +import org.apache.flink.runtime.state.CheckpointStorageAccess; +import org.apache.flink.runtime.state.ConfigurableCheckpointStorage; +import org.apache.flink.runtime.state.memory.MemoryBackendCheckpointStorageAccess; + +import java.io.IOException; + +import static org.apache.flink.util.Preconditions.checkArgument; + +/** + * The {@link CheckpointStorage} checkpoints state directly to the JobManager's memory (hence the + * name), but savepoints will be persisted to a file system. + * + * <p>This checkpoint storage is primarily for experimentation, quick local setups, or for streaming + * applications that have very small state: Because it requires checkpoints to go through the + * JobManager's memory, larger state will occupy larger portions of the JobManager's main memory, + * reducing operational stability. For any other setup, the {@link FileSystemCheckpointStorage} + * should be used. The {@code FileSystemCheckpointStorage} but checkpoints state directly to files + * rather than to the JobManager's memory, thus supporting larger state sizes and more highly + * available recovery. + * + * <h1>State Size Considerations</h1> + * + * <p>State checkpointing with this storage is subject to the following conditions: + * + * <ul> + * <li>Each individual state must not exceed the configured maximum state size (see {@link + * #getMaxStateSize()}. + * <li>All state from one task (i.e., the sum of all operator states and keyed states from all + * chained operators of the task) must not exceed what the RPC system supports, which is be + * default < 10 MB. That limit can be configured up, but that is typically not advised. + * <li>The sum of all states in the application times all retained checkpoints must comfortably + * fit into the JobManager's JVM heap space. + * </ul> + * + * <h1>Persistence Guarantees</h1> + * + * <p>For the use cases where the state sizes can be handled by this checkpoint storage, the storage + * does guarantee persistence for savepoints, externalized checkpoints (if configured), and + * checkpoints (when high-availability is configured). + * + * <h1>Configuration</h1> + * + * <p>As for all checkpoint storage, this storage policy can either be configured within the + * application (by creating the storage with the respective constructor parameters and setting it on + * the execution environment) or by specifying it in the Flink configuration. + * + * <p>If the checkpoint storage was specified in the application, it may pick up additional + * configuration parameters from the Flink configuration. For example, if the storage if configured + * in the application without a default savepoint directory, it will pick up a default savepoint + * directory specified in the Flink configuration of the running job/cluster. That behavior is + * implemented via the {@link #configure(ReadableConfig, ClassLoader)} method. + */ +@PublicEvolving +public class JobManagerCheckpointStorage extends AbstractFileCheckpointStorage + implements CheckpointStorage, ConfigurableCheckpointStorage { Review comment: That's fair, I'll rework this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
