Hi Denis, Only WAL disk is corrupted but datastore is intact by any way can we restore the cluster ? some data loss is fine. Any suggestion on this?
Thanks and Regards, Kamlesh Joshi From: Kamlesh Joshi Sent: Thursday, June 6, 2019 7:52 PM To: [email protected] Subject: RE: [External]Re: Is there any way to force recover the cluster - copying running cluster datastore Thanks for the update Denis. If one of the WAL disk gets failed, is there any way to start or recover the cluster forcefully ? Thanks and Regards, Kamlesh Joshi From: Denis Magda <[email protected]<mailto:[email protected]>> Sent: Thursday, June 6, 2019 4:44 PM To: [email protected]<mailto:[email protected]> Subject: [External]Re: Is there any way to force recover the cluster - copying running cluster datastore The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. I would discourage you from doing this if data consistency is prominent for you. What you see on the disk of one cluster node might be inconsistent with the whole cluster state and the actual/last updates in memory. Snapshots and backups can solve your task. Google for a solution provided by GridGain. - Denis On Wed, Jun 5, 2019 at 8:27 AM Kamlesh Joshi <[email protected]<mailto:[email protected]>> wrote: Hi Team, We are trying to start another Ignite cluster by taking a copy of the running cluster’s datastore (source cluster’s datastore is getting modified in parallel). So, when we try to start the server node with copied datastore, it gives error as below. Also, giving cluster configuration for reference: pageSize=#{4 * 1024} walMode=LOG_ONLY walFlushFrequency=60000 rebalanceThreadPoolSize=8 rebalanceThrottle=100 rebalanceBatchSize=#{32 * 1024 * 1024} storagePath=/datastore/datastore walPath=/datastore1/wal walArchivePath=/datastore1/archive metadataWorkDir=/datastore/metadataWorkDir [2019-06-05T12:21:52,943][INFO ][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null] [2019-06-05T12:21:52,967][INFO ][main][PageMemoryImpl] Started page memory [memoryAllocated=128.0 MiB, pages=31744, tableSize=2.5 MiB, checkpointBuffer=100.0 MiB] [2019-06-05T12:21:52,968][INFO ][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [2019-06-05T12:21:52,973][ERROR][main][IgniteKernal%EDIFCustomer_DR] Exception during start processors, node will be stopped and close connections org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1742) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:980) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.Ignition.start(Ignition.java:352) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) [ignite-core-2.6.0.jar:2.6.0] Caused by: org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@4d6c2], start=FileWALPointer [idx=0, fileOff=0, len=0]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:3009) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2960) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2896) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:799) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1968) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:574) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:525) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:700) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739) ~[ignite-core-2.6.0.jar:2.6.0] ... 11 more [2019-06-05T12:21:52,978][ERROR][main][IgniteKernal%EDIFCustomer_DR] Got exception while starting (will rollback startup routine). org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1742) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:980) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.Ignition.start(Ignition.java:352) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) [ignite-core-2.6.0.jar:2.6.0] Caused by: org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@4d6c2], start=FileWALPointer [idx=0, fileOff=0, len=0]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:3009) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2960) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2896) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:799) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1968) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:574) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:525) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:700) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739) ~[ignite-core-2.6.0.jar:2.6.0] So, is there any way to start this cluster with copied data store forcefully? This scenario may also arrive if WAL disk gets failed. How can we atleast start the cluster with minimum data loss ? Any help would be highly appreciated ! Thanks and Regards, Kamlesh Joshi "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
