Found the below exception with further analysis of the logs :
2020-08-16 13:34:14,306 ERROR o.a.i.l.l.Log4J2Logger
[jdbc-cache-loader-#10199%ig_svc_cluster%] Critical system error detected.
Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=795360926,
val2=844420635324779]], cacheId=-1799995498, cacheName=T_NAME,
indexName=FOO_NAMESEARCH_BRN_DESC_IDX, msg=Runtime failure on row:
Row@6a714d32[ key: 311822, val: foo.TName [...REDACTED...] ][
...REDACTED... ]]]]
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=795360926,
val2=844420635324779]], cacheId=-1799995498, cacheName=T_NAME,
indexName=FOO_NAMESEARCH_BRN_DESC_IDX, msg=Runtime failure on row:
Row@6a714d32[ key: 311822, val: foo.TName [...REDACTED...] ][...REDACTED...
]]
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.corruptedTreeException(H2Tree.java:673)
[ignite-indexing-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2380)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2327)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:428)
[ignite-indexing-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:844)
[ignite-indexing-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:782)
[ignite-indexing-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:380)
[ignite-indexing-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2079)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:409)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2627)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1713)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1688)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2444)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:445)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4285)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3426)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:807)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:772)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.loadEntry(GridDhtCacheAdapter.java:690)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.access$600(GridDhtCacheAdapter.java:104)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:640)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:636)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter$3.apply(GridCacheStoreManagerAdapter.java:535)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:470)
[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:434)
[ignite-core-2.8.1.jar!/:2.8.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.lang.IllegalStateException: Duplicate row in index.
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:442)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:428)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5711)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5697)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:360)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:297)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11300(BPlusTree.java:94)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:3681)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7100(BPlusTree.java:3361)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2625)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2606)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2606)
~[ignite-core-2.8.1.jar!/:2.8.1]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2347)
[ignite-core-2.8.1.jar!/:2.8.1]
... 27 more
I also observe a bug with the same exception being fixed recently as part
of https://issues.apache.org/jira/browse/IGNITE-10873
Are there any known reasons for the trigger of " Caused by:
java.lang.IllegalStateException: Duplicate row in index. " ?
On Sat, Aug 15, 2020 at 1:56 PM Srikanta Patanjali <[email protected]>
wrote:
> Resharing the cache settings as it got snipped in the previous email:
>
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="clientMode" value="true"/>
> <property name="igniteInstanceName" value="ig_svc_cluster"/>
> <property name="metricsLogFrequency" value="0"/>
>
> <property name="discoverySpi">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
> <property name="addresses">
> <list>
> <value>127.0.0.1:47500..47510</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
> <!-- Enabling Apache Ignite native persistence. -->
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="name" value="Default_Region"/>
> <!-- 500 MB initial size. -->
> <property name="initialSize" value="#{500L * 1024
> * 1024}"/>
> <!-- Setting the size of the default region to
> 1GB. -->
> <property name="maxSize" value="#{1L * 1024 * 1024
> * 1024}"/>
> </bean>
> </property>
> <property name="dataRegionConfigurations">
> <list>
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="name"
> value="FooNamesCache_Region"/>
> <!-- 500 MB initial size. -->
> <property name="initialSize" value="#{500L *
> 1024 * 1024}"/>
> <!-- Setting the size of the default region to
> 2GB. -->
> <property name="maxSize" value="#{2L * 1024 *
> 1024 * 1024}"/>
> <property name="persistenceEnabled"
> value="true"/>
> </bean>
> </list>
> </property>
> </bean>
> </property>
>
> <property name="cacheConfiguration">
> <list>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="dataRegionName"
> value="FooNamesCache_Region"/>
> <property name="name" value="foo_TNameCache"/>
> <property name="cacheMode" value="REPLICATED"/>
> <property name="atomicityMode" value="ATOMIC"/>
> <property name="copyOnRead" value="true"/>
> <property name="onheapCacheEnabled" value="true"/>
> <property name="eagerTtl" value="true"/>
>
> <property name="evictionPolicyFactory">
> <bean
> class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory">
> <property name="batchSize" value="200"/>
> <property name="maxMemorySize" value="100000"/>
> </bean>
> </property>
>
> <property name="sqlOnheapCacheEnabled" value="true"/>
> <property name="sqlOnheapCacheMaxSize" value="100"/>
>
> <property name="cacheStoreFactory">
> <bean
> class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
> <property name="dataSourceBean"
> value="dsPostgreSQL_FooNames"/>
> <property name="dialect">
> <bean
> class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect">
> </bean>
> </property>
>
> <property name="types">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcType">
> <property name="cacheName"
> value="foo_TNameCache"/>
> <property name="keyType"
> value="java.lang.Integer"/>
> <property name="valueType"
> value="foo.TName"/>
> <property name="databaseSchema"
> value="public"/>
> <property name="databaseTable"
> value="t_name"/>
>
> <property name="keyFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.INTEGER"/>
> </constructor-arg>
> <constructor-arg
> value="id"/>
> <constructor-arg
> value="int"/>
> <constructor-arg
> value="id"/>
> </bean>
> </list>
> </property>
>
> <property name="valueFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.TIMESTAMP"/>
> </constructor-arg>
> <constructor-arg
> value="date"/>
> <constructor-arg
> value="java.sql.Timestamp"/>
> <constructor-arg
> value="date"/>
> </bean>
>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.TIMESTAMP"/>
> </constructor-arg>
> <constructor-arg
> value="modified_date"/>
> <constructor-arg
> value="java.sql.Timestamp"/>
> <constructor-arg
> value="modified_date"/>
> </bean>
>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.NUMERIC"/>
> </constructor-arg>
> <constructor-arg
> value="secondary_id"/>
> <constructor-arg
> value="java.math.BigDecimal"/>
> <constructor-arg
> value="secondary_id"/>
> </bean>
>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.VARCHAR"/>
> </constructor-arg>
> <constructor-arg
> value="name_orig"/>
> <constructor-arg
> value="java.lang.String"/>
> <constructor-arg
> value="nameOrig"/>
> </bean>
>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg>
> <util:constant
> static-field="java.sql.Types.VARCHAR"/>
> </constructor-arg>
> <constructor-arg
> value="name_search"/>
> <constructor-arg
> value="java.lang.String"/>
> <constructor-arg
> value="nameSearch"/>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </property>
>
> <property name="readThrough" value="true"/>
> <property name="writeThrough" value="true"/>
>
> <property name="queryEntities">
> <list>
> <bean
> class="org.apache.ignite.cache.QueryEntity">
> <property name="keyType"
> value="java.lang.Integer"/>
> <property name="valueType"
> value="foo.TName"/>
> <property name="tableName" value="t_name"/>
> <property name="keyFieldName" value="id"/>
>
> <property name="keyFields">
> <list>
> <value>id</value>
> </list>
> </property>
>
> <property name="fields">
> <map>
> <entry key="secondary_id"
> value="java.math.BigDecimal"/>
> <entry key="nameSearch"
> value="java.lang.String"/>
> <entry key="id"
> value="java.lang.Integer"/>
> </map>
> </property>
>
> <property name="aliases">
> <map>
> <entry key="nameOrig"
> value="name_orig"/>
> <entry key="nameSearch"
> value="name_search"/>
> </map>
> </property>
>
> <!-- Defining indexed fields.-->
> <property name="indexes">
> <list>
> <!-- Single field (aka. column)
> index -->
> <bean
> class="org.apache.ignite.cache.QueryIndex">
> <constructor-arg
> value="nameSearch"/>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
>
>
> On Sat, Aug 15, 2020 at 1:32 PM p.srikanta <[email protected]> wrote:
>
>> I've been consistently getting a error "class
>> o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree
>> is
>> corrupted".
>>
>> I know a related bug was recently fixed in v2.8
>> (https://issues.apache.org/jira/browse/IGNITE-12593) and another one is
>> in
>> progress (https://issues.apache.org/jira/browse/IGNITE-12911. However I'm
>> not sure if my scenario is related to either one of them (though it is
>> close
>> to IGNITE-12593).
>>
>> Sharing more details to seek help in either identifying the bug or
>> suggestion for possible work around, thanks in advance!
>>
>> *Note*: The below excpetion was not observed when the mentioned cache was
>> initialized without any indexed field i.e without any QueryIndex within
>> the
>> QueryEntity config.
>>
>> *Ignite Version*: 2.8.1
>>
>> *Encountered Exception*
>>
>>
>> *Size of the DB table*: 25M rows
>>
>> *Cache Settings*
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
