Thanks for the quick response. I have observed similar behavior with 3rd party persistence read through IF I set indexed types for the cache.
Test case - Load up the cache using put with 35,000 entries ( keys 1 -> 35,000). Read every key using Get(key) This is the use case that I want to use in my application, where I have an active subset of my data in memory and if a key is accessed that is not in memory, it is read in from a postgres database. I am only doing read through, not write through since there is a different path for the data to get into postgres. I can see from cache metrics ( shown below ) that I perform 34011 reads, of which there are 6093 hits and 27,918 misses and then I get the OOM error. This only happens if indexed types are set on the cache. Is this expected behavior? If I am not using sql query on the cache , only using get and put, does it matter if I do not set indexedTypes? Does it help or hurt performance in any way? Cache metrics and stack trace shown below. CacheMetricsSnapshot [reads=34011, puts=35000, hits=6093, misses=27918, txCommits=0, txRollbacks=0, evicts=0, removes=0, putAvgTimeNanos=190.56557, getAvgTimeNanos=53.07989, rmvAvgTimeNanos=0.0, commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=fubar, offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0, offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=33726, heapEntriesCnt=2, offHeapPrimaryEntriesCnt=33726, offHeapBackupEntriesCnt=0, offHeapAllocatedSize=0, size=33726, keySize=33726, isEmpty=false, dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0, txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0, txCommittedVersionsSize=0, txRolledbackVersionsSize=0, txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0, txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0, txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1, isWriteBehindEnabled=false, writeBehindFlushSize=-1, writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1, writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1, writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1, writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0, keysToRebalanceLeft=0, rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0, rebalanceFinishTime=0, keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true, isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=true, isWriteThrough=false] [14:46:05,572][ERROR][sys-#52][GridPartitionedSingleGetFuture] Failed to get values from dht cache [fut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: Not enough memory allocated (consider increasing data region size or enabling evictions) [policyName=RefData, size=22.0 MB], hash=315679498]] class org.apache.ignite.IgniteCheckedException: Not enough memory allocated (consider increasing data region size or enabling evictions) [policyName=RefData, size=22.0 MB] at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7252) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:975) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Not enough memory allocated (consider increasing data region size or enabling evictions) [policyName=RefData, size=22.0 MB] at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePageNoReuse(DataStructure.java:117) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePage(DataStructure.java:105) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8400(BPlusTree.java:81) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insertWithSplit(BPlusTree.java:2703) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insert(BPlusTree.java:2665) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$2500(BPlusTree.java:2547) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:411) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:392) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:342) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:261) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11100(BPlusTree.java:81) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:2859) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7600(BPlusTree.java:2547) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2285) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:1977) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.put(H2TreeIndex.java:197) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:537) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:488) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:559) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1747) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:425) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1354) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1209) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:343) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3191) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.versionedValue(GridCacheMapEntry.java:2726) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16$1.apply(GridCacheAdapter.java:2032) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16$1.apply(GridCacheAdapter.java:2011) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:423) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:389) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2011) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2009) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1412) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6631) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967) On Wed, Jan 10, 2018 at 8:46 AM, Alexey Popov <[email protected]> wrote: > Hi, > > You are right, cache.putAll() can't evict the entries from the batch it is > working on, and you can get Ignite OOME. > This is expected behavior because putAll get locks for all provided entry > keys. That is critical: > 1) for transactional caches and > 2) any caches backed up by 3-rd party persistence store. > > There was an intention to optimize this behavior for atomic caches without > cache store [1] but it seems it will not be implemented. So, you could rely > on this behavior. > > [1] https://issues.apache.org/jira/browse/IGNITE-514. > > Thank you, > Alexey > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
