Glad to hear that you got it working.
> On 30 Jan 2023, at 23:05, Łukasz Dywicki <l...@code-house.org> wrote:
>
> Hello Stephen,
> I've started tests with few angles, with larger and smaller memory setups,
> looked at Ignite unit tests and began looking for page eviction triggers.
> Once I found interesting code areas I realized that my test setup was not
> having proper page eviction set up. This lead me to further debugging which
> lead me piece of old code which was overriding persistence configuration,
> *effectively* reverting page eviction to default (no eviction).
> After clean up I successfully tested eviction with small caches (512MB max +
> direct buffer at 756MB). These worked stable under constant load, which
> proved the point that it is possible to have stable operation on fairly
> limited resources.
>
> Guilt goes to spring configuration pre/post-processing which went under radar
> till today. It was really specific to test application itself, not Ignite. I
> thank you for feedback, pointers and assistance!
>
> Kind regards,
> Łukasz
>
> On 30.01.2023 12:51, Stephen Darlington wrote:
>> I’m not saying you should test with 50Gb, but I am saying that 512Mb is too
>> small. Maybe try a few gigs.
>> When you define eviction at a cache level, that is for the on-heap cache.
>> Most people don’t need an on-heap cache.
>> You you define eviction at a data region level, that works on the off-heap
>> data. As you observe, it works at a /page/ rather than a record level. (This
>> is why a tiny region can be problematic. There are far few pages that can be
>> chosen to evict.)
>>> On 30 Jan 2023, at 11:37, Łukasz Dywicki <l...@code-house.org> wrote:
>>>
>>> Dear Jeremy and Stephen,
>>> Thank you for answers. I this issues I face is not bound exclusively to
>>> assigned memory but behavior of eviction policies. After all, without
>>> proper eviction or expiry policy, each amount of memory will eventually be
>>> exceeded. Small size of memory makes it simply easier to test. To be honest
>>> I do not see a point in testing eviction with 50GB of memory, if it cant be
>>> configured for 512MB. Expiry policy proved to work, but it will not
>>> preventing node from crash, as there always might be high write ratio
>>> causing memory to fill.
>>>
>>> Both LRU and FIFO eviction policies in default configuration do not take
>>> into account overall memory allocation, they work only with on heap caches.
>>> Unless memory size is explicitly specified they will rely on
>>> over-provisioned memory, or assume that memory assignment is right. This is
>>> possible only if cache entry have a predictable.
>>> Another point is that above eviction policies are dedicated to on-heap
>>> caches and are told to have no affect to off-heap region, why is so? By
>>> looking at sources I see CacheOffheapEvictionManager and
>>> GridCacheEvictionManager, but I can't grasp what enables eviction of
>>> entries from off heap memory managed by Ignite.
>>> Given that my use case is continuous operation with variety of entries
>>> having various sizes I can't assume on-heap cache with LRU nor FIFO,
>>> especially that they do not impact off heap memory. I also can't risk
>>> filling in memory managed by Ignite due to lack of clear eviction policy
>>> during peak load.
>>> As far I can understand from DataRegionConfiguration, its eviction
>>> mechanism works for whole pages, but I haven't seen this happening. Could
>>> it be due to fact that single cache entry in my test exceeds maximum page
>>> size (16MB)?
>>>
>>> Best regards,
>>> Łukasz
>>>
>>> On 30.01.2023 10:32, Stephen Darlington wrote:
>>>> Ignite is designed to work as a cluster with a lot of memory. A single
>>>> node with 512Mb of memory just isn’t what most people are testing with.
>>>> I’ve seen similar issues when people use tiny nodes. I don’t see anything
>>>> immediately wrong with your configuration, but if it works with more
>>>> memory, that’s your problem.
>>>>> On 28 Jan 2023, at 00:35, Łukasz Dywicki <l...@code-house.org> wrote:
>>>>>
>>>>> Hello again,
>>>>> I've spent another day on this issue looking at configuration and with
>>>>> cluster larger than configured backup count ignite did the work.
>>>>> After that I reverted to testing of single node with minimal
>>>>> configuration and come to the point that the only way to keep Ignite
>>>>> survive load was setting ExpiryPolicy to very low value (10s).
>>>>>
>>>>> To me it seems that Ignite behavior is to preserve all entries in memory
>>>>> at any cost, even if there is a risk of running into OOM. Is that true?
>>>>> I would like to change this behavior and make sure that entries do not
>>>>> expire because of time as long as there is memory available to store
>>>>> them. They should be evicted by appropriate EvictionPolicy only if memory
>>>>> fills up.
>>>>> To me it looks like DataStoreSettings do not make any impact in this
>>>>> regard. At least setting page eviction to LRU do not change it.
>>>>>
>>>>> Please let me know if I am doing something wrong as I can not prove
>>>>> Ignite to be working stable. Even with such basic objectives I outlined
>>>>> in earlier mail.
>>>>>
>>>>> Kind regards,
>>>>> Łukasz
>>>>>
>>>>> On 27.01.2023 00:58, Łukasz Dywicki wrote:
>>>>>> Dear all,
>>>>>> I come across use of Apache Ignite to cache results of expensive
>>>>>> computation operation.
>>>>>> Objectives are basic:
>>>>>> - Keep most of "hot" data in memory
>>>>>> - Offload cold part to cache store
>>>>>> - Keep memory utilization under control (evict entries as needed)
>>>>>> While it sounds basic, it doesn't seem to fit Ignite defaults.
>>>>>> What I am testing now is behavior with large objects which can grow up
>>>>>> to 10 mb (serialized) or 25 mb (json representation). Usually objects
>>>>>> will stay far below that threshold, but we can't make assumption on that.
>>>>>> I began testing various configurations of Ignite in order to facilitate
>>>>>> offloading of memory contents to database. So far I am stuck for two
>>>>>> days at Ignite/application itself running out of memory after processing
>>>>>> several of such large objects. While I know that storing 10 mb blob in
>>>>>> database is not the best idea, I have to test that behavior too.
>>>>>> By observing database contents I see that number of entries there grows,
>>>>>> but cache do not seem to be evicted. When I try to switch eviction, it
>>>>>> does require onheap to be switched on, and it still fails with LRU
>>>>>> eviction policy.
>>>>>> So far I ended up with a named cache and default region configured as
>>>>>> below:
>>>>>> ```
>>>>>> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>>>>>> igniteConfiguration.setDataStorageConfiguration(new
>>>>>> DataStorageConfiguration()
>>>>>> .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
>>>>>> .setPersistenceEnabled(false)
>>>>>> .setInitialSize(256L * 1024 * 1024)
>>>>>> .setMaxSize(512L * 1024 * 1024)
>>>>>> .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
>>>>>> .setSwapPath(null)
>>>>>> .setEvictionThreshold(0.75)
>>>>>> )
>>>>>> .setPageSize(DataStorageConfiguration.MAX_PAGE_SIZE)
>>>>>> );
>>>>>> CacheConfiguration<SessionKey, ExpensiveObject> expensiveCache = new
>>>>>> CacheConfiguration<>()
>>>>>> .setName(CACHE_NAME)
>>>>>> .setBackups(2)
>>>>>> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
>>>>>> .setCacheStoreFactory(cacheJdbcBlobStoreFactory)
>>>>>> .setWriteThrough(true)
>>>>>> .setOnheapCacheEnabled(true)
>>>>>> .setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1024))
>>>>>> .setReadThrough(true);
>>>>>> igniteConfiguration.setCacheConfiguration(
>>>>>> expensiveCache
>>>>>> );
>>>>>> ```
>>>>>> What I observe is following - the cache keeps writing data into
>>>>>> database, but it does not remove old entries fast enough to prevent
>>>>>> crash.
>>>>>> JVM parameters I use are fairly basic:
>>>>>> -Xms1g -Xmx1g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
>>>>>> -XX:+DisableExplicitGC
>>>>>> The store mechanism is jdbc blob store. Exceptions I get happen to occur
>>>>>> in Ignite itself, processing (application code writing cache) or
>>>>>> communication thread used to feed cache. I collected one case here:
>>>>>> https://gist.github.com/splatch/b5ec9134cd9df19bc62f007dd17a19a1
>>>>>> The error message in linked gist advice to enable persistence (which I
>>>>>> did via cache store!), increase memory limit (which I don't want to do),
>>>>>> or enable eviction/expiry policy (which somehow miss behave).
>>>>>> To me it looks like self defense mechanisms Ignite has are being tricked
>>>>>> leading whole application to crash.
>>>>>> Can you please advise me which settings to tune and how in order to get
>>>>>> Ignite more stable under such load?
>>>>>> Kind regards,
>>>>>> Łukasz
