Joel Lang created IGNITE-6506:
---------------------------------
Summary: Unable to activate cluster node if it was stopped during
persistent storage checkpoint
Key: IGNITE-6506
URL: https://issues.apache.org/jira/browse/IGNITE-6506
Project: Ignite
Issue Type: Bug
Components: persistence
Affects Versions: 2.2
Reporter: Joel Lang
Priority: Critical
I have a cluster with two nodes: A and B.
On startup, node A and B wait for each other to be connected and then node A
will attempt to activate the cluster.
While testing high availability we find that if a node is stopped during the
persistent store checkpoint, we cannot activate the cluster on startup without
deleting the persistent storage directory. Specifically in the case where node
A is stopped during checkpointing, upon the next startup it will encounter
several exceptions during activation and then hang without completing
activation.
Here is the log.
{noformat}
2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO
o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.ignite.internal.exchange.time - Started exchange init
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18,
node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990,
addrs=[10.5.17.19, 127.0.0.1],
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true,
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], evtNode=TcpDiscoveryNode
[id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1],
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true,
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false],
customEvt=ChangeGlobalStateMessage
[id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7,
reqId=f7155dea-fede-4340-b244-7a3b65f167a8,
initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process
[nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false,
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory:
/opt/mbe1/ignite/db/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work
directory: /opt/mbe1/ignite/db/wal/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive
directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default
MemoryPolicy found; system default of 1GB size will be used.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory
[memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB,
checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=3.1
GiB, pages=1544064, tableSize=91.0 MiB, checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start
marker =
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin,
end marker =
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state
[lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731,
forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0,
forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the
middle of checkpoint. Will restore memory state and finish checkpoint on node
start.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node
components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false,
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
java.lang.ArrayIndexOutOfBoundsException: -1
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92)
at
org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=1], waitTime=0ms]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start
marker =
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin,
end marker =
/opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates
since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0,
len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize
local partitions (preloading will be stopped): GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb,
evt=DISCOVERY_CUSTOM_EVT]
org.apache.ignite.IgniteCheckedException: WAL history is too short
[descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization
completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion
of partition map exchange (preloading will not start):
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1],
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true,
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2, nodeId8=62cf0ccb,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1506445884063]],
crd=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990,
addrs=[10.5.17.19, 127.0.0.1],
sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true,
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false,
hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], exchActions=null, affChangeMsg=null,
skipPreload=false, clientOnlyExchange=false, initTs=1506445884073,
centralizedAff=false,
changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1,
forcedRebFut=null, done=true, evtLatch=0,
remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException:
WAL history is too short
[descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]],
hash=927462353]]
org.apache.ignite.IgniteCheckedException: WAL history is too short
[descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
