Hi Ignitians, I fail to understand what causes and need your help - 1) When k8s sees “Critical system error”, it will restart ignite-admin server. Restarting is fine because of the critical system error. But what are the causes of the critical system error? 2) Critical system error may be corresponding to JVM held. Still we don’t know the reason why JVM got held. 3) The cluster lost one ignite client node, probably due to OOME 4) Why/how was ignite server node triggered to reload the data from Cassandra (all data in C* tables are cached once the ignite server starts. All SQL DML interacts with Ignite Cache which interact with Cassandra for insert/update/delete) If Ignite needs to rebalance the data among the server nodes, why can't rebalance the data from one node to another? if even rebalancing data, why submitting invalid queries? We are using apache-ignite-2.8.0.20190215.
Exceptions - [2021-06-02 17:09:04,005][ERROR][sys-#103562%ignite-procurant-admin-cluster%][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception]] ... 1 more at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:104) class org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:800) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:847) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:795) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.salvageTx(GridDistributedTxRemoteAdapter.java:898) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.salvageTx(IgniteTxManager.java:398) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.access$3100(IgniteTxManager.java:134) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject.onTimeout0(IgniteTxManager.java:2551) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject.access$3300(IgniteTxManager.java:2505) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$1.run(IgniteTxManager.java:2624) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6898) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING at java.lang.Thread.run(Thread.java:745) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:293) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:338) Caused by: class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteException: Failed to execute Cassandra CQL statement: select "id", "sourceid", "versioning", "colname", "colnewvalue", "cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where "id"=?; at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerReload(GridCacheMapEntry.java:984) ... 13 more at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.readThrough(GridCacheMapEntry.java:619) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:701) ... 18 more at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:198) Caused by: class org.apache.ignite.IgniteException: Failed to execute Cassandra CQL statement: select "id", "sourceid", "versioning", "colname", "colnewvalue", "cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where "id"=?; Caused by: javax.cache.integration.CacheLoaderException: class org.apache.ignite.IgniteException: Failed to execute Cassandra CQL statement: select "id", "sourceid", "versioning", "colname", "colnewvalue", "cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where "id"=?; ... 17 more at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:189) Caused by: class org.apache.ignite.IgniteException: Failed to execute Cassandra CQL statement: select "id", "sourceid", "versioning", "colname", "colnewvalue", "cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where "id"=?; at org.apache.ignite.cache.store.cassandra.session.WrappedSession.prepare(WrappedSession.java:56) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:180) ... 20 more at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.prepareStatement(CassandraSessionImpl.java:665) Caused by: class org.apache.ignite.IgniteException: Failed to prepare Cassandra CQL statement: select "id", "sourceid", "versioning", "colname", "colnewvalue", "cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where "id"=?; at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:147) at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:104) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.cache.store.cassandra.session.WrappedSession.prepare(WrappedSession.java:56) ... 21 more Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:327) at com.datastax.driver.core.Responses$Error.asException(Responses.java:148) at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:196) at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:220) at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442) at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433) at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408) at com.google.common.util.concurrent.Futures$2$1.run(Futures.java:1177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [2021-06-02 17:09:04,556][ERROR][sys-#103562%ignite-procurant-admin-cluster%][root] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception]] ... 1 more at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.prepareStatement(CassandraSessionImpl.java:647) ... 20 more [17:09:02] Topology snapshot [ver=7, locNode=103dfcb4, servers=2, clients=3, state=ACTIVE, CPUs=15, offheap=6.5GB, heap=8.0GB] -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
