Alexey, I'm wondering you had a chance to look into this? I'd like to understand what to expect in terms of node activation time and how it's related to the data volume.
Thanks! Andrey ________________________________ From: Andrey Kornev <[email protected]> Sent: Monday, January 22, 2018 11:36 AM To: Alexey Goncharuk; [email protected] Subject: Re: Long activation times with Ignite persistence enabled Alexey, Thanks a lot for looking into this! My configuration is very basic: 3 caches all using standard 1024 partitions, sharing a 1GB persistent memory region. Please find below the stack trace of the exchange worker thread captured while the node's activation is in progress (2.4 Ignite branch). Hope it helps! Thanks! Andrey "exchange-worker-#42%ignite-2%" #82 prio=5 os_prio=31 tid=0x00007ffe8bf1c000 nid=0xc403 waiting on condition [0x000070000ed43000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145) at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:324) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:306) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:291) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:656) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:576) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:130) at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:212) at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:367) at org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.<init>(CacheFreeListImpl.java:47) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore$1.<init>(GridCacheOffheapManager.java:1041) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1041) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1247) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:835) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.casState(GridDhtLocalPartition.java:523) - locked <0x000000077a3d1120> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:218) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.forceCreatePartition(GridDhtPartitionTopologyImpl.java:804) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restorePartitionState(GridCacheDatabaseSharedManager.java:2196) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2155) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1322) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1113) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1063) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:661) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2329) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) 2018-01-22 11:30:01,049 INFO [exchange-worker-#42%ContentStore-2%] GridCacheDatabaseSharedManager - Finished applying WAL changes [updatesApplied=0, time=68435ms] 2018-01-22 11:30:01,789 INFO [main] GridDiscoveryManager - Topology snapshot [ver=4, servers=2, clients=0, CPUs=8, offheap=26.0GB, heap=4.0GB] 2018-01-22 11:30:01,789 INFO [main] GridDiscoveryManager - Data Regions Configured: 2018-01-22 11:30:01,789 INFO [main] GridDiscoveryManager - ^-- default [initSize=256.0 MiB, maxSize=12.0 GiB, persistenceEnabled=false] 2018-01-22 11:30:01,789 INFO [main] GridDiscoveryManager - ^-- durable [initSize=256.0 MiB, maxSize=1.0 GiB, persistenceEnabled=true]
