Hello, Maxim. Great to see such an important feature in Ignite! Please, let me know if you need any help with review.
> 7 апр. 2020 г., в 22:10, Maxim Muzafarov <maxmu...@gmail.com> написал(а): > > Igniters, > > > I'd like to back to the discussion of a snapshot operation for Apache > Ignite for persistence cache groups and I propose my changes below. I > have prepared everything so that the discussion is as meaningful and > specific as much as possible: > > - IEP-43: Cluster snapshot [1] > - The Jira task IGNITE-11073 [2] > - PR with described changes, Patch Available [4] > > Changes are ready for review. > > > Here are a few implementation details and my thoughts: > > 1. Snapshot restore assumed to be manual at the first step. The > process will be described on our documentation pages, but it is > possible to start node right from the snapshot directory since the > directory structure is preserved (check > `testConsistentClusterSnapshotUnderLoad` in the PR). We also have some > options here about how the restore process must look like: > - fully manual snapshot restore (will be documented) > - ansible or shell scripts for restore > - Java API for restore (I doubt we should go this way). > > 3. The snapshot `create` procedure creates a snapshot of all > persistent caches available on the cluster (see limitations [1]). > > 2. The snapshot `create` procedure is available through Java API and > JMX (control.sh may be implemented further). > > Java API: > IgniteFuture<Void> fut = ignite.snapshot() > .createSnapshot(name); > > JMX: > SnapshotMXBean mxBean = getMBean(ignite.name()); > mxBean.createSnapshot(name); > > 3. The Distribute Process [3] is used to perform a cluster-wide > snapshot procedure, so we've avoided a lot of boilerplate code here. > > 4. The design document [1] contains also an internal API for creating > a consistent local snapshot of requested cache groups and transfer it > to another node using the FileTransmission protocol [6]. This is one > of the parts of IEP-28 [5] for cluster rebalancing via partition files > and an important part for understanding the whole design. > > Java API: > public IgniteInternalFuture<Void> createRemoteSnapshot( > UUID rmtNodeId, > Map<Integer, Set<Integer>> parts, > BiConsumer<File, GroupPartitionId> partConsumer); > > > Please, share your thoughts and take a loot at my changes [4]. > > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-43%3A+Cluster+snapshots > [2] https://issues.apache.org/jira/browse/IGNITE-11073 > [3] > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/util/distributed/DistributedProcess.java#L49 > [4] https://github.com/apache/ignite/pull/7607 > [5] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-Filetransferbetweennodes > [6] > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/communication/TransmissionHandler.java#L42 > > > On Thu, 28 Feb 2019 at 14:43, Dmitriy Pavlov <dpav...@apache.org> wrote: >> >> Hi Maxim, >> >> I agree with Denis and I have just one concern here. >> >> Apache Ignite has quite a long story (started even before Apache), and now >> it has a way too huge number of features. Some of these features >> - are developed and well known by community members, >> - some of them were contributed a long time ago and nobody develops it, >> - and, actually, in some rare cases, nobody in the community knows how it >> works and how to change it. >> >> Such features may attract users, but a bug in it may ruin impression about >> the product. Even worse, nobody can help to solve it, and only user himself >> or herself may be encouraged to contribute a fix. >> >> And my concern here, such a big feature should have a number of interested >> contributors, who can support it in case if others lost interest. I will be >> happy if 3-5 members will come and say, yes, I will do a review/I will help >> with further changes. >> >> Just to be clear, I'm not against it, and I'll never cast -1 for it, but it >> would be more comfortable to develop this feature with understanding that >> this work will not be useless. >> >> Sincerely, >> Dmitriy Pavlov >> >> ср, 27 февр. 2019 г. в 23:36, Denis Magda <dma...@apache.org>: >> >>> Maxim, >>> >>> GridGain has this exact feature available for Ignite native persistence >>> deployments. It's not as easy as it might have been seen from the >>> enablement perspective. Took us many years to make it production ready, >>> involving many engineers. If the rest of the community wants to create >>> something similar and available in open source then please take this >>> estimate into consideration. >>> >>> - >>> Denis >>> >>> >>> On Wed, Feb 27, 2019 at 8:53 AM Maxim Muzafarov <maxmu...@gmail.com> >>> wrote: >>> >>>> Igniters, >>>> >>>> Some of the stores with which the Apache Ignite is often compared has >>>> a feature called Snapshots [1] [2]. This feature provides an >>>> eventually consistent view on stored data for different purposes (e.g. >>>> moving data between environments, saving a backup of data for the >>>> further restore procedure and so on). The Apache Ignite has all >>>> opportunities and machinery to provide cache and\or data region >>>> snapshots out of the box but still don't have them. >>>> >>>> This issue derives from IEP-28 [5] on which I'm currently working on >>>> (partially described in the section [6]). I would like to solve this >>>> issue too and make Apache Ignite more attractive to use on a >>>> production environment. I've haven't investigated in-memory type >>>> caches yet, but for caches with enabled persistence, we can do it >>>> without any performance impact on cache operations (some additional IO >>>> operations are needed to copy cache data to backup store, copy on >>>> write technique is used here). We just need to use our DiscoverySpi, >>>> PME and Checkpointer process the right way. >>>> >>>> For the first step, we can store all backup data on each of cache >>>> affinity node locally. For instance, the `backup\snapshotId\cache0` >>>> folder will be created and all `cache0` partitions will be stored >>>> there for each local node for the snapshot process with id >>>> `snapshotId`. In future, we can teach nodes to upload snapshotted >>>> partitions to the one remote node or cloud. >>>> >>>> -- >>>> >>>> High-level process overview >>>> >>>> A new snapshot process is managed via DiscoverySpi and >>>> CommunicationSpi messages. >>>> >>>> 1. The initiator sends a request to the cluster (DiscoveryMessage). >>>> 2. When the node receives a message it initiates PME. >>>> 3. The node begins checkpoint process (holding write lock a short time) >>>> 4. The node starts to track any write attempts to the snapshotting >>>> partition and places the copy of original pages to the temp file. >>>> 5. The node performs merge the partition file with the corresponding >>> delta. >>>> 6. When the node finishes the backup process it sends ack message with >>>> saved partitions to the initiator (or the error response). >>>> 7. When all ack messages received the backup is finished. >>>> >>>> The only problem here is that when the request message arrives at the >>>> particular node during running checkpoint PME will be locked until it >>>> ends. This is not good. But hopefully, it will be fixed here [4]. >>>> >>>> -- >>>> >>>> Probable API >>>> >>>> From the cache perspective: >>>> >>>> IgniteFuture<IgniteSnapshot> snapshotFut = >>>> ignite.cache("default") >>>> .shapshotter() >>>> .create("myShapshotId"); >>>> >>>> IgniteSnapshot cacheSnapshot = snapshotFut.get(); >>>> >>>> IgniteCache<K, V> copiedCache = >>>> ignite.createCache("CopyCache") >>>> .withConfiguration(defaultCache.getConfiguration()) >>>> .loadFromSnapshot(cacheSnapshot.id()); >>>> >>>> From the command line perspective: >>>> >>>> control.sh --snapshot take cache0,cache1,cache2 >>>> >>>> -- >>>> >>>> WDYT? >>>> Will it be a useful feature for the Apache Ignite? >>>> >>>> >>>> [1] >>>> >>> https://geode.apache.org/docs/guide/10/managing/cache_snapshots/chapter_overview.html >>>> [2] >>>> >>> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsBackupTakesSnapshot.html >>>> [3] >>>> >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-Snapshots-in-Ignite-td4183.html >>>> [4] https://issues.apache.org/jira/browse/IGNITE-10508 >>>> [5] >>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing >>>> [6] >>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-Checkpointer >>>> >>>
