I’m not saying you should test with 50Gb, but I am saying that 512Mb is too small. Maybe try a few gigs.
When you define eviction at a cache level, that is for the on-heap cache. Most people don’t need an on-heap cache. You you define eviction at a data region level, that works on the off-heap data. As you observe, it works at a page rather than a record level. (This is why a tiny region can be problematic. There are far few pages that can be chosen to evict.) > On 30 Jan 2023, at 11:37, Łukasz Dywicki <l...@code-house.org> wrote: > > Dear Jeremy and Stephen, > Thank you for answers. I this issues I face is not bound exclusively to > assigned memory but behavior of eviction policies. After all, without proper > eviction or expiry policy, each amount of memory will eventually be exceeded. > Small size of memory makes it simply easier to test. To be honest I do not > see a point in testing eviction with 50GB of memory, if it cant be configured > for 512MB. Expiry policy proved to work, but it will not preventing node from > crash, as there always might be high write ratio causing memory to fill. > > Both LRU and FIFO eviction policies in default configuration do not take into > account overall memory allocation, they work only with on heap caches. Unless > memory size is explicitly specified they will rely on over-provisioned > memory, or assume that memory assignment is right. This is possible only if > cache entry have a predictable. > Another point is that above eviction policies are dedicated to on-heap caches > and are told to have no affect to off-heap region, why is so? By looking at > sources I see CacheOffheapEvictionManager and GridCacheEvictionManager, but I > can't grasp what enables eviction of entries from off heap memory managed by > Ignite. > Given that my use case is continuous operation with variety of entries having > various sizes I can't assume on-heap cache with LRU nor FIFO, especially that > they do not impact off heap memory. I also can't risk filling in memory > managed by Ignite due to lack of clear eviction policy during peak load. > As far I can understand from DataRegionConfiguration, its eviction mechanism > works for whole pages, but I haven't seen this happening. Could it be due to > fact that single cache entry in my test exceeds maximum page size (16MB)? > > Best regards, > Łukasz > > On 30.01.2023 10:32, Stephen Darlington wrote: >> Ignite is designed to work as a cluster with a lot of memory. A single node >> with 512Mb of memory just isn’t what most people are testing with. I’ve seen >> similar issues when people use tiny nodes. I don’t see anything immediately >> wrong with your configuration, but if it works with more memory, that’s your >> problem. >>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <l...@code-house.org> wrote: >>> >>> Hello again, >>> I've spent another day on this issue looking at configuration and with >>> cluster larger than configured backup count ignite did the work. >>> After that I reverted to testing of single node with minimal configuration >>> and come to the point that the only way to keep Ignite survive load was >>> setting ExpiryPolicy to very low value (10s). >>> >>> To me it seems that Ignite behavior is to preserve all entries in memory at >>> any cost, even if there is a risk of running into OOM. Is that true? >>> I would like to change this behavior and make sure that entries do not >>> expire because of time as long as there is memory available to store them. >>> They should be evicted by appropriate EvictionPolicy only if memory fills >>> up. >>> To me it looks like DataStoreSettings do not make any impact in this >>> regard. At least setting page eviction to LRU do not change it. >>> >>> Please let me know if I am doing something wrong as I can not prove Ignite >>> to be working stable. Even with such basic objectives I outlined in earlier >>> mail. >>> >>> Kind regards, >>> Łukasz >>> >>> On 27.01.2023 00:58, Łukasz Dywicki wrote: >>>> Dear all, >>>> I come across use of Apache Ignite to cache results of expensive >>>> computation operation. >>>> Objectives are basic: >>>> - Keep most of "hot" data in memory >>>> - Offload cold part to cache store >>>> - Keep memory utilization under control (evict entries as needed) >>>> While it sounds basic, it doesn't seem to fit Ignite defaults. >>>> What I am testing now is behavior with large objects which can grow up to >>>> 10 mb (serialized) or 25 mb (json representation). Usually objects will >>>> stay far below that threshold, but we can't make assumption on that. >>>> I began testing various configurations of Ignite in order to facilitate >>>> offloading of memory contents to database. So far I am stuck for two days >>>> at Ignite/application itself running out of memory after processing >>>> several of such large objects. While I know that storing 10 mb blob in >>>> database is not the best idea, I have to test that behavior too. >>>> By observing database contents I see that number of entries there grows, >>>> but cache do not seem to be evicted. When I try to switch eviction, it >>>> does require onheap to be switched on, and it still fails with LRU >>>> eviction policy. >>>> So far I ended up with a named cache and default region configured as >>>> below: >>>> ``` >>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); >>>> igniteConfiguration.setDataStorageConfiguration(new >>>> DataStorageConfiguration() >>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration() >>>> .setPersistenceEnabled(false) >>>> .setInitialSize(256L * 1024 * 1024) >>>> .setMaxSize(512L * 1024 * 1024) >>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU) >>>> .setSwapPath(null) >>>> .setEvictionThreshold(0.75) >>>> ) >>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE) >>>> ); >>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new >>>> CacheConfiguration<>() >>>> .setName(CACHE_NAME) >>>> .setBackups(2) >>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC) >>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory) >>>> .setWriteThrough(true) >>>> .setOnheapCacheEnabled(true) >>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024)) >>>> .setReadThrough(true); >>>> igniteConfiguration.setCacheConfiguration( >>>> expensiveCache >>>> ); >>>> ``` >>>> What I observe is following - the cache keeps writing data into database, >>>> but it does not remove old entries fast enough to prevent crash. >>>> JVM parameters I use are fairly basic: >>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC >>>> -XX:+DisableExplicitGC >>>> The store mechanism is jdbc blob store. Exceptions I get happen to occur >>>> in Ignite itself, processing (application code writing cache) or >>>> communication thread used to feed cache. I collected one case here: >>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1 >>>> The error message in linked gist advice to enable persistence (which I did >>>> via cache store!), increase memory limit (which I don't want to do), or >>>> enable eviction/expiry policy (which somehow miss behave). >>>> To me it looks like self defense mechanisms Ignite has are being tricked >>>> leading whole application to crash. >>>> Can you please advise me which settings to tune and how in order to get >>>> Ignite more stable under such load? >>>> Kind regards, >>>> Łukasz
