Vyacheslav, In this case community should definitely take a look and investigate. Please share your results when you have a chance.
-Val On Wed, Dec 6, 2017 at 1:45 AM, Vyacheslav Daradur <daradu...@gmail.com> wrote: > Evgeniy, as far as I understand PDS and rebalancing are based on > page-memory approach instead of entry-based 3rd Party Persistence, so > I'm not sure how to extend rebalancing behavior properly. > > Dmitry, the performance is the only reason of why I try to solve > rebalancing issue. > I've benchmarked RocksDB as 3rd party persistence and PDS via Ignite > Yardstick with "fsync" enabled in both cases. > The result shows that PDS is twice slower on "put" operation on the > single node, but I had had no time to do benchmarks on all sides. > I'll try to do that next week and will share results if the community > is interested. Maybe there will be no reason for using RocksDB. > > > > On Fri, Nov 24, 2017 at 4:58 PM, Dmitry Pavlov <dpavlov....@gmail.com> > wrote: > > Please see the discussion on the user list. It seems that the same > happened > > there: > > > > http://apache-ignite-users.70518.x6.nabble.com/Reassign- > partitions-td7461.html#a7468 > > > > it contains examples when the data can diverge. > > > > пт, 24 нояб. 2017 г. в 16:42, Dmitry Pavlov <dpavlov....@gmail.com>: > > > >> If we compare native and 3rd party persistence (cache store): > >> - Updating and reading data from DBMS is slower in most scenarios. > >> - Non-clustered DBMS is a single point of failure, it is hard to scale. > >> - Ignite SQL does not extend to External (3rd party persitsence) Cache > >> Store (and queries ignore DBMS changes). > >> > >> > >> Which is why I am wondering if Native persistence is applicable in this > >> case decribed by Vyacheslav. > >> > >> пт, 24 нояб. 2017 г. в 12:23, Evgeniy Ignatiev < > >> yevgeniy.ignat...@gmail.com>: > >> > >>> Sorry linked the wrong page, the latter url is not the example. > >>> > >>> > >>> On 11/24/2017 1:12 PM, Evgeniy Ignatiev wrote: > >>> > By the way I remembered that there is an annotation CacheLocalStore > >>> > for marking exactly the CacheStore that is not distributed - > >>> > > >>> http://apache-ignite-developers.2346864.n4.nabble. > com/CacheLocalStore-td734.html > >>> > - here is short explanation and this - > >>> > > >>> https://github.com/gridgain/gridgain-advanced-examples/ > blob/master/src/main/java/org/gridgain/examples/localstore/ > LocalRecoverableStoreExample.java > >>> > - is example implementation. > >>> > > >>> > > >>> > On 11/23/2017 4:42 PM, Dmitry Pavlov wrote: > >>> >> Hi Evgeniy, > >>> >> > >>> >> Technically it is, of course, possible, but still > >>> >> - it is not simple at all > >>> >> - IgniteCacheOffheapManager & IgniteWriteAheadLogManager are > internal > >>> >> APIs, > >>> >> and community can change any APIs here in any time. > >>> >> > >>> >> Vyacheslav, > >>> >> > >>> >> Why Ignite Native Persistence is not suitable for this case? > >>> >> > >>> >> Sincerely, > >>> >> Dmitriy Pavlov > >>> >> > >>> >> чт, 23 нояб. 2017 г. в 11:01, Evgeniy Ignatiev > >>> >> <yevgeniy.ignat...@gmail.com > >>> >>> : > >>> >>> As far as I remember, last webinar I heard on Ignite Native > >>> Persistence > >>> >>> - it actually exposes some interfaces like > IgniteWriteAheadLogManager, > >>> >>> PageStore, PageStoreManager, etc., with the file-based > implementation > >>> >>> provided by Ignite being only one possible approach, and users can > >>> >>> create their own Native Persistence variations. At least that what > has > >>> >>> been said by Denis Magda at that time. > >>> >>> > >>> >>> May be creating own implementation of Ignite Native Persistence > rather > >>> >>> than CacheStore based persistence is an option here? > >>> >>> > >>> >>> On 11/23/2017 2:23 AM, Valentin Kulichenko wrote: > >>> >>>> Vyacheslav, > >>> >>>> > >>> >>>> There is no way to do this and I'm not sure why you want to do > this. > >>> >>> Ignite > >>> >>>> persistence was developed to solve exactly the problems you're > >>> >>> describing. > >>> >>>> Just use it :) > >>> >>>> > >>> >>>> -Val > >>> >>>> > >>> >>>> On Wed, Nov 22, 2017 at 12:36 AM, Vyacheslav Daradur < > >>> >>> daradu...@gmail.com> > >>> >>>> wrote: > >>> >>>> > >>> >>>>> Valentin, Evgeniy thanks for your help! > >>> >>>>> > >>> >>>>> Valentin, unfortunately, you are right. > >>> >>>>> > >>> >>>>> I've tested that behavior in the following scenario: > >>> >>>>> 1. Started N nodes and filled it with data > >>> >>>>> 2. Shutdown one node > >>> >>>>> 3. Called rebalance directly and waited to finish > >>> >>>>> 4. Stopped all other (N-1) nodes > >>> >>>>> 5. Started N-1 nodes and validated data > >>> >>>>> > >>> >>>>> Validation didn't pass - data consistency was broken. As you say > it > >>> >>>>> works only on stable topology. > >>> >>>>> As far as I understand Ignite doesn't manage to rebalance in > >>> >>>>> underlying storage, it became clear from tests and your > description > >>> >>>>> that CacheStore design assumes that the underlying storage is > shared > >>> >>>>> by all the > >>> >>>>> nodes in the topology. > >>> >>>>> > >>> >>>>> I understand that PDS is the best option in case of distributing > >>> >>>>> persistence. > >>> >>>>> However, could you point me the best way to override default > >>> >>>>> rebalance > >>> >>>>> behavior? > >>> >>>>> Maybe it's possible to extend it by a custom plugin? > >>> >>>>> > >>> >>>>> On Wed, Nov 22, 2017 at 1:35 AM, Valentin Kulichenko > >>> >>>>> <valentin.kuliche...@gmail.com> wrote: > >>> >>>>>> Vyacheslav, > >>> >>>>>> > >>> >>>>>> If you want the persistence storage to be *distributed*, then > using > >>> >>>>> Ignite > >>> >>>>>> persistence would be the easiest thing to do anyway, even if you > >>> >>>>>> don't > >>> >>>>> need > >>> >>>>>> all its features. > >>> >>>>>> > >>> >>>>>> CacheStore indeed can be updated from different nodes with > >>> different > >>> >>>>> nodes, > >>> >>>>>> but the problem is in coordination. If instances of the store > are > >>> >>>>>> not > >>> >>>>> aware > >>> >>>>>> of each other, it's really hard to handle all rebalancing cases. > >>> >>>>>> Such > >>> >>>>>> solution will work only on stable topology. > >>> >>>>>> > >>> >>>>>> Having said that, if you can have one instance of RocksDB (or > any > >>> >>>>>> other > >>> >>>>> DB > >>> >>>>>> for that matter) that is accessed via network by all nodes, then > >>> >>>>>> it's > >>> >>>>> also > >>> >>>>>> an option. But in this case storage is not distributed. > >>> >>>>>> > >>> >>>>>> -Val > >>> >>>>>> > >>> >>>>>> On Tue, Nov 21, 2017 at 4:37 AM, Vyacheslav Daradur < > >>> >>> daradu...@gmail.com > >>> >>>>>> wrote: > >>> >>>>>> > >>> >>>>>>> Valentin, > >>> >>>>>>> > >>> >>>>>>>>> Why don't you use Ignite persistence [1]? > >>> >>>>>>> I have a use case for one of the projects that need the RAM on > >>> disk > >>> >>>>>>> replication only. All PDS features aren't needed. > >>> >>>>>>> During the first assessment, persist to RocksDB works faster. > >>> >>>>>>> > >>> >>>>>>>>> CacheStore design assumes that the underlying storage is > >>> >>>>>>>>> shared by > >>> >>>>> all > >>> >>>>>>> the nodes in topology. > >>> >>>>>>> This is the very important note. > >>> >>>>>>> I'm a bit confused because I've thought that each node in > cluster > >>> >>>>>>> persists partitions for which the node is either primary or > backup > >>> >>>>>>> like in PDS. > >>> >>>>>>> > >>> >>>>>>> My RocksDB implementation supports working with one DB instance > >>> >>>>>>> which > >>> >>>>>>> shared by all the nodes in the topology, but it would make no > >>> >>>>>>> sense of > >>> >>>>>>> using embedded fast storage. > >>> >>>>>>> > >>> >>>>>>> Is there any link to a detailed description of CacheStorage > >>> >>>>>>> design or > >>> >>>>>>> any other advice? > >>> >>>>>>> Thanks in advance. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> On Fri, Nov 17, 2017 at 9:07 PM, Valentin Kulichenko > >>> >>>>>>> <valentin.kuliche...@gmail.com> wrote: > >>> >>>>>>>> Vyacheslav, > >>> >>>>>>>> > >>> >>>>>>>> CacheStore design assumes that the underlying storage is > shared > >>> by > >>> >>> all > >>> >>>>>>> the > >>> >>>>>>>> nodes in topology. Even if you delay rebalancing on node stop > >>> >>>>>>>> (which > >>> >>>>> is > >>> >>>>>>>> possible via CacheConfiguration#rebalanceDelay), I doubt it > will > >>> >>>>> solve > >>> >>>>>>> all > >>> >>>>>>>> your consistency issues. > >>> >>>>>>>> > >>> >>>>>>>> Why don't you use Ignite persistence [1]? > >>> >>>>>>>> > >>> >>>>>>>> [1] > >>> >>>>>>>> https://apacheignite.readme.io/docs/distributed- > persistent-store > >>> >>>>>>>> > >>> >>>>>>>> -Val > >>> >>>>>>>> > >>> >>>>>>>> On Fri, Nov 17, 2017 at 4:24 AM, Vyacheslav Daradur < > >>> >>>>> daradu...@gmail.com > >>> >>>>>>>> wrote: > >>> >>>>>>>> > >>> >>>>>>>>> Hi Andrey! Thank you for answering. > >>> >>>>>>>>> > >>> >>>>>>>>>>> Key to partition mapping shouldn't depends on topology, and > >>> >>>>> shouldn't > >>> >>>>>>>>> changed unstable topology. > >>> >>>>>>>>> Key to partition mapping doesn't depend on topology in my > test > >>> >>>>>>>>> affinity function. It only depends on partitions number. > >>> >>>>>>>>> But partition to node mapping depends on topology and at > cluster > >>> >>>>> stop, > >>> >>>>>>>>> when one node left topology, some partitions may be moved to > >>> >>>>>>>>> other > >>> >>>>>>>>> nodes. > >>> >>>>>>>>> > >>> >>>>>>>>>>> Does all nodes share same RockDB database or each node has > >>> >>>>>>>>>>> its own > >>> >>>>>>> copy? > >>> >>>>>>>>> Each Ignite node has own RocksDB instance. > >>> >>>>>>>>> > >>> >>>>>>>>>>> Would you please share configuration? > >>> >>>>>>>>> It's pretty simple: > >>> >>>>>>>>> IgniteConfiguration cfg = new > IgniteConfiguration(); > >>> >>>>>>>>> cfg.setIgniteInstanceName(instanceName); > >>> >>>>>>>>> > >>> >>>>>>>>> CacheConfiguration<Integer, String> cacheCfg = new > >>> >>>>>>>>> CacheConfiguration<>(); > >>> >>>>>>>>> cacheCfg.setName(TEST_CACHE_NAME); > >>> >>>>>>>>> cacheCfg.setCacheMode(CacheMode.PARTITIONED); > >>> >>>>>>>>> cacheCfg.setWriteSynchronizationMode( > >>> >>>>>>>>> CacheWriteSynchronizationMode.PRIMARY_SYNC); > >>> >>>>>>>>> cacheCfg.setBackups(1); > >>> >>>>>>>>> cacheCfg.setAffinity(new > >>> >>>>>>>>> TestAffinityFunction(partitionsNumber, backupsNumber)); > >>> >>>>>>>>> cacheCfg.setWriteThrough(true); > >>> >>>>>>>>> cacheCfg.setReadThrough(true); > >>> >>>>>>>>> cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC); > >>> >>>>>>>>> cacheCfg.setCacheStoreFactory(new > >>> >>>>>>>>> RocksDBCacheStoreFactory<>("/test/path/to/persistence", > >>> >>>>>>>>> TEST_CACHE_NAME, cfg)); > >>> >>>>>>>>> > >>> >>>>>>>>> cfg.setCacheConfiguration(cacheCfg); > >>> >>>>>>>>> > >>> >>>>>>>>> Could you give me advice on places which I need to pay > >>> attention? > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> On Wed, Nov 15, 2017 at 3:02 PM, Andrey Mashenkov > >>> >>>>>>>>> <andrey.mashen...@gmail.com> wrote: > >>> >>>>>>>>>> Hi Vyacheslav, > >>> >>>>>>>>>> > >>> >>>>>>>>>> Key to partition mapping shouldn't depends on topology, and > >>> >>>>> shouldn't > >>> >>>>>>>>>> changed unstable topology. > >>> >>>>>>>>>> Looks like you've missed smth. > >>> >>>>>>>>>> > >>> >>>>>>>>>> Would you please share configuration? > >>> >>>>>>>>>> Does all nodes share same RockDB database or each node has > >>> >>>>>>>>>> its own > >>> >>>>>>> copy? > >>> >>>>>>>>>> > >>> >>>>>>>>>> On Wed, Nov 15, 2017 at 12:22 AM, Vyacheslav Daradur < > >>> >>>>>>>>> daradu...@gmail.com> > >>> >>>>>>>>>> wrote: > >>> >>>>>>>>>> > >>> >>>>>>>>>>> Hi, Igniters! > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> I’m using partitioned Ignite cache with RocksDB as 3rd > party > >>> >>>>>>> persistence > >>> >>>>>>>>>>> store. > >>> >>>>>>>>>>> I've got an issue: if cache rebalancing is switched on, > then > >>> >>>>>>>>>>> it’s > >>> >>>>>>>>>>> possible to lose some data. > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> Basic scenario: > >>> >>>>>>>>>>> 1) Start Ignite cluster and fill a cache with RocksDB > >>> >>>>>>>>>>> persistence; > >>> >>>>>>>>>>> 2) Stop all nodes > >>> >>>>>>>>>>> 3) Start Ignite cluster and validate data > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> This works fine while rebalancing is switched off. > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> If rebalancing switched on: when I call Ignition#stopAll, > some > >>> >>>>> nodes > >>> >>>>>>>>>>> go down sequentially and while one node having gone down > >>> >>>>>>>>>>> another > >>> >>>>>>> start > >>> >>>>>>>>>>> rebalancing. When nodes started affinity function works > with a > >>> >>>>> full > >>> >>>>>>>>>>> set of nodes and may define a wrong partition for a key > >>> because > >>> >>>>> the > >>> >>>>>>>>>>> previous state was changed at rebalancing. > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> Maybe I'm doing something wrong. How can I avoid > rebalancing > >>> >>>>>>>>>>> while > >>> >>>>>>>>>>> stopping all nodes in the cluster? > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> Could you give me any advice, please? > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> -- > >>> >>>>>>>>>>> Best Regards, Vyacheslav D. > >>> >>>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> -- > >>> >>>>>>>>>> Best regards, > >>> >>>>>>>>>> Andrey V. Mashenkov > >>> >>>>>>>>> > >>> >>>>>>>>> -- > >>> >>>>>>>>> Best Regards, Vyacheslav D. > >>> >>>>>>>>> > >>> >>>>>>> > >>> >>>>>>> -- > >>> >>>>>>> Best Regards, Vyacheslav D. > >>> >>>>>>> > >>> >>>>> > >>> >>>>> -- > >>> >>>>> Best Regards, Vyacheslav D. > >>> >>>>> > >>> >>> > >>> > > >>> > >>> > > > > -- > Best Regards, Vyacheslav D. >
