Hi Maxim,
I agree with Denis and I have just one concern here.
Apache Ignite has quite a long story (started even before Apache), and now
it has a way too huge number of features. Some of these features
- are developed and well known by community members,
- some of them were contributed a long time ago and nobody develops it,
- and, actually, in some rare cases, nobody in the community knows how it
works and how to change it.
Such features may attract users, but a bug in it may ruin impression about
the product. Even worse, nobody can help to solve it, and only user himself
or herself may be encouraged to contribute a fix.
And my concern here, such a big feature should have a number of interested
contributors, who can support it in case if others lost interest. I will be
happy if 3-5 members will come and say, yes, I will do a review/I will help
with further changes.
Just to be clear, I'm not against it, and I'll never cast -1 for it, but it
would be more comfortable to develop this feature with understanding that
this work will not be useless.
Sincerely,
Dmitriy Pavlov
ср, 27 февр. 2019 г. в 23:36, Denis Magda <dma...@apache.org>:
> Maxim,
>
> GridGain has this exact feature available for Ignite native persistence
> deployments. It's not as easy as it might have been seen from the
> enablement perspective. Took us many years to make it production ready,
> involving many engineers. If the rest of the community wants to create
> something similar and available in open source then please take this
> estimate into consideration.
>
> -
> Denis
>
>
> On Wed, Feb 27, 2019 at 8:53 AM Maxim Muzafarov <maxmu...@gmail.com>
> wrote:
>
> > Igniters,
> >
> > Some of the stores with which the Apache Ignite is often compared has
> > a feature called Snapshots [1] [2]. This feature provides an
> > eventually consistent view on stored data for different purposes (e.g.
> > moving data between environments, saving a backup of data for the
> > further restore procedure and so on). The Apache Ignite has all
> > opportunities and machinery to provide cache and\or data region
> > snapshots out of the box but still don't have them.
> >
> > This issue derives from IEP-28 [5] on which I'm currently working on
> > (partially described in the section [6]). I would like to solve this
> > issue too and make Apache Ignite more attractive to use on a
> > production environment. I've haven't investigated in-memory type
> > caches yet, but for caches with enabled persistence, we can do it
> > without any performance impact on cache operations (some additional IO
> > operations are needed to copy cache data to backup store, copy on
> > write technique is used here). We just need to use our DiscoverySpi,
> > PME and Checkpointer process the right way.
> >
> > For the first step, we can store all backup data on each of cache
> > affinity node locally. For instance, the `backup\snapshotId\cache0`
> > folder will be created and all `cache0` partitions will be stored
> > there for each local node for the snapshot process with id
> > `snapshotId`. In future, we can teach nodes to upload snapshotted
> > partitions to the one remote node or cloud.
> >
> > --
> >
> > High-level process overview
> >
> > A new snapshot process is managed via DiscoverySpi and
> > CommunicationSpi messages.
> >
> > 1. The initiator sends a request to the cluster (DiscoveryMessage).
> > 2. When the node receives a message it initiates PME.
> > 3. The node begins checkpoint process (holding write lock a short time)
> > 4. The node starts to track any write attempts to the snapshotting
> > partition and places the copy of original pages to the temp file.
> > 5. The node performs merge the partition file with the corresponding
> delta.
> > 6. When the node finishes the backup process it sends ack message with
> > saved partitions to the initiator (or the error response).
> > 7. When all ack messages received the backup is finished.
> >
> > The only problem here is that when the request message arrives at the
> > particular node during running checkpoint PME will be locked until it
> > ends. This is not good. But hopefully, it will be fixed here [4].
> >
> > --
> >
> > Probable API
> >
> > From the cache perspective:
> >
> > IgniteFuture<IgniteSnapshot> snapshotFut =
> > ignite.cache("default")
> > .shapshotter()
> > .create("myShapshotId");
> >
> > IgniteSnapshot cacheSnapshot = snapshotFut.get();
> >
> > IgniteCache<K, V> copiedCache =
> > ignite.createCache("CopyCache")
> > .withConfiguration(defaultCache.getConfiguration())
> > .loadFromSnapshot(cacheSnapshot.id());
> >
> > From the command line perspective:
> >
> > control.sh --snapshot take cache0,cache1,cache2
> >
> > --
> >
> > WDYT?
> > Will it be a useful feature for the Apache Ignite?
> >
> >
> > [1]
> >
> https://geode.apache.org/docs/guide/10/managing/cache_snapshots/chapter_overview.html
> > [2]
> >
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsBackupTakesSnapshot.html
> > [3]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Data-Snapshots-in-Ignite-td4183.html
> > [4] https://issues.apache.org/jira/browse/IGNITE-10508
> > [5]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing
> > [6]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-Checkpointer
> >
>
