[
https://issues.apache.org/jira/browse/IGNITE-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Pavlov updated IGNITE-8742:
-----------------------------------
Description:
https://ci.ignite.apache.org/viewLog.html?buildId=1366882&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_PdsDirectIo2
Test
org.apache.ignite.internal.processors.cache.persistence.IgniteNativeIoWalFlushFsyncSelfTest#testFailAfterStart
emulates problem with disc space using exception.
In direct IO environment real IO with disk is performed, tmpfs is not used.
Sometimes this error can come from rollover() of segment, failure handler
reacted accordingly.
{noformat}
detected. Will be handled accordingly to configured handler [hnd=class
o.a.i.failure.StopNodeFailureHandler, failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.pagemem.wal.StorageException: Unable to
write]]
class org.apache.ignite.internal.pagemem.wal.StorageException: Unable to write
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.writeBuffer(FsyncModeFileWriteAheadLogManager.java:2964)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2640)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2572)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2525)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.close(FsyncModeFileWriteAheadLogManager.java:2795)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$700(FsyncModeFileWriteAheadLogManager.java:2340)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.rollOver(FsyncModeFileWriteAheadLogManager.java:1029)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.log(FsyncModeFileWriteAheadLogManager.java:673)
{noformat}
But test seems to be not able to stop, node stopper thread tries to stop cache,
flush WAL. flush wait for rollover, which will never happen.
{noformat}
Thread [name="node-stopper", id=2836, state=WAITING, blockCnt=7, waitCnt=9]
Lock
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f6473,
ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
at o.a.i.i.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7473)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2546)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.fsync(FsyncModeFileWriteAheadLogManager.java:2750)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$2000(FsyncModeFileWriteAheadLogManager.java:2340)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.flush(FsyncModeFileWriteAheadLogManager.java:699)
at
o.a.i.i.processors.cache.GridCacheProcessor.stopCache(GridCacheProcessor.java:1243)
at
o.a.i.i.processors.cache.GridCacheProcessor.stopCaches(GridCacheProcessor.java:969)
at
o.a.i.i.processors.cache.GridCacheProcessor.stop(GridCacheProcessor.java:943)
at o.a.i.i.IgniteKernal.stop0(IgniteKernal.java:2289)
at o.a.i.i.IgniteKernal.stop(IgniteKernal.java:2167)
at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2588)
- locked o.a.i.i.IgnitionEx$IgniteNamedInstance@90f6bfd
at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2551)
at o.a.i.i.IgnitionEx.stop(IgnitionEx.java:372)
at
o.a.i.failure.StopNodeFailureHandler$1.run(StopNodeFailureHandler.java:36)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It seems invalidating environment of WAL manager is not working propertly.
was:
Test
org.apache.ignite.internal.processors.cache.persistence.IgniteNativeIoWalFlushFsyncSelfTest#testFailAfterStart
emulates problem with disc space using exception.
In direct IO environment real IO with disk is performed, tmpfs is not used.
Sometimes this error can come from rollover() of segment, failure handler
reacted accordingly.
{noformat}
detected. Will be handled accordingly to configured handler [hnd=class
o.a.i.failure.StopNodeFailureHandler, failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.pagemem.wal.StorageException: Unable to
write]]
class org.apache.ignite.internal.pagemem.wal.StorageException: Unable to write
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.writeBuffer(FsyncModeFileWriteAheadLogManager.java:2964)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2640)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2572)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2525)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.close(FsyncModeFileWriteAheadLogManager.java:2795)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$700(FsyncModeFileWriteAheadLogManager.java:2340)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.rollOver(FsyncModeFileWriteAheadLogManager.java:1029)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.log(FsyncModeFileWriteAheadLogManager.java:673)
{noformat}
But test seems to be not able to stop, node stopper thread tries to stop cache,
flush WAL. flush wait for rollover, which will never happen.
{noformat}
Thread [name="node-stopper", id=2836, state=WAITING, blockCnt=7, waitCnt=9]
Lock
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f6473,
ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
at o.a.i.i.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7473)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2546)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.fsync(FsyncModeFileWriteAheadLogManager.java:2750)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$2000(FsyncModeFileWriteAheadLogManager.java:2340)
at
o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.flush(FsyncModeFileWriteAheadLogManager.java:699)
at
o.a.i.i.processors.cache.GridCacheProcessor.stopCache(GridCacheProcessor.java:1243)
at
o.a.i.i.processors.cache.GridCacheProcessor.stopCaches(GridCacheProcessor.java:969)
at
o.a.i.i.processors.cache.GridCacheProcessor.stop(GridCacheProcessor.java:943)
at o.a.i.i.IgniteKernal.stop0(IgniteKernal.java:2289)
at o.a.i.i.IgniteKernal.stop(IgniteKernal.java:2167)
at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2588)
- locked o.a.i.i.IgnitionEx$IgniteNamedInstance@90f6bfd
at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2551)
at o.a.i.i.IgnitionEx.stop(IgnitionEx.java:372)
at
o.a.i.failure.StopNodeFailureHandler$1.run(StopNodeFailureHandler.java:36)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It seems invalidating environment of WAL manager is not working propertly.
> Direct IO 2 suite is timed out by 'out of disk space' failure emulation test:
> WAL manager failure does not stoped execution
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-8742
> URL: https://issues.apache.org/jira/browse/IGNITE-8742
> Project: Ignite
> Issue Type: Test
> Components: persistence
> Reporter: Dmitriy Pavlov
> Priority: Major
> Labels: MakeTeamcityGreenAgain
>
> https://ci.ignite.apache.org/viewLog.html?buildId=1366882&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_PdsDirectIo2
> Test
> org.apache.ignite.internal.processors.cache.persistence.IgniteNativeIoWalFlushFsyncSelfTest#testFailAfterStart
> emulates problem with disc space using exception.
> In direct IO environment real IO with disk is performed, tmpfs is not used.
> Sometimes this error can come from rollover() of segment, failure handler
> reacted accordingly.
> {noformat}
> detected. Will be handled accordingly to configured handler [hnd=class
> o.a.i.failure.StopNodeFailureHandler, failureCtx=FailureContext
> [type=CRITICAL_ERROR, err=class o.a.i.i.pagemem.wal.StorageException: Unable
> to write]]
> class org.apache.ignite.internal.pagemem.wal.StorageException: Unable to write
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.writeBuffer(FsyncModeFileWriteAheadLogManager.java:2964)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2640)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flush(FsyncModeFileWriteAheadLogManager.java:2572)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2525)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.close(FsyncModeFileWriteAheadLogManager.java:2795)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$700(FsyncModeFileWriteAheadLogManager.java:2340)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.rollOver(FsyncModeFileWriteAheadLogManager.java:1029)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.log(FsyncModeFileWriteAheadLogManager.java:673)
> {noformat}
> But test seems to be not able to stop, node stopper thread tries to stop
> cache, flush WAL. flush wait for rollover, which will never happen.
> {noformat}
> Thread [name="node-stopper", id=2836, state=WAITING, blockCnt=7, waitCnt=9]
> Lock
> [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f6473,
> ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
> at o.a.i.i.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7473)
> at
> o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.flushOrWait(FsyncModeFileWriteAheadLogManager.java:2546)
> at
> o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.fsync(FsyncModeFileWriteAheadLogManager.java:2750)
> at
> o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager$FileWriteHandle.access$2000(FsyncModeFileWriteAheadLogManager.java:2340)
> at
> o.a.i.i.processors.cache.persistence.wal.FsyncModeFileWriteAheadLogManager.flush(FsyncModeFileWriteAheadLogManager.java:699)
> at
> o.a.i.i.processors.cache.GridCacheProcessor.stopCache(GridCacheProcessor.java:1243)
> at
> o.a.i.i.processors.cache.GridCacheProcessor.stopCaches(GridCacheProcessor.java:969)
> at
> o.a.i.i.processors.cache.GridCacheProcessor.stop(GridCacheProcessor.java:943)
> at o.a.i.i.IgniteKernal.stop0(IgniteKernal.java:2289)
> at o.a.i.i.IgniteKernal.stop(IgniteKernal.java:2167)
> at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2588)
> - locked o.a.i.i.IgnitionEx$IgniteNamedInstance@90f6bfd
> at o.a.i.i.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2551)
> at o.a.i.i.IgnitionEx.stop(IgnitionEx.java:372)
> at
> o.a.i.failure.StopNodeFailureHandler$1.run(StopNodeFailureHandler.java:36)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> It seems invalidating environment of WAL manager is not working propertly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
