Thanks folks. I’ll improve the doc by 2.4 release: IGNITE-7645 - Clarify eviction policy documentation <https://issues.apache.org/jira/browse/IGNITE-7645>
— Denis > On Feb 6, 2018, at 11:25 PM, Alexey Goncharuk <alexey.goncha...@gmail.com> > wrote: > > Denis, > > Yes, if there are no concurrent updates and there are no ongoing > transactions preventing the entries to be evicted, then the whole page will > be purged. > > 2018-02-07 0:23 GMT+03:00 Denis Magda <dma...@apache.org>: > >> Alexey, >> >> Let me understand this part because it’s still not crystal clear to me: >> >>> Now as per the >>> eviction mechanism, it is both per-page and per-entry: first, we choose a >>> page which fits most for eviction, however, we cannot simply erase the >> page >>> because quite a lot of data structures are referring to that page. To >> deal >>> with it, we read keys that the chosen page contains and then clear >>> individual cache entries, which in turn will clear up all necessary >> links. >> >> >> However will all the keys-values from a chosen page be removed eventually >> during the eviction phase? If it’s true then it’s still a page based >> eviction - we select 5 random oldest pages and purge all the key-values >> from them. >> >> — >> Denis >> >>> On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk < >> alexey.goncha...@gmail.com> wrote: >>> >>> Guys, >>> >>> To clarify the questions, I would like to reiterate over the mechanics of >>> evictions and then suggest a renaming that should resolve such things in >>> the future. >>> >>> First of all, Lucas is right - eviction policy only makes sense when >> native >>> persistence is disabled because what it does is actually freeing up >> memory >>> when a user hits the memory limit. The only way to do this is to destroy >>> inserted data because there is no other way to free memory. Now as per >> the >>> eviction mechanism, it is both per-page and per-entry: first, we choose a >>> page which fits most for eviction, however, we cannot simply erase the >> page >>> because quite a lot of data structures are referring to that page. To >> deal >>> with it, we read keys that the chosen page contains and then clear >>> individual cache entries, which in turn will clear up all necessary >> links. >>> If there are no concurrent updates, the page becomes empty and will be >>> reused for other user data. This is exactly what is explained on the wiki >>> page (at least in my reading of the page). >>> >>> Second, at this point, I would rename page management mechanism with >>> enabled persistence from 'eviction' to 'page replacement' or 'page >>> swapping', because it does not destroy any data, but rather replaces a >>> chosen buffer in memory from one page to another. The content of neither >>> pages does not change during page replacement. This mechanism is unlikely >>> to be selected by a user because the effectiveness of page replacements >>> heavily depends on internal data structures and may change from version >> to >>> version, and may even be adaptive depending on the load pattern. >>> >>> Hope this resolves the confusion. >>> >>> --AG >>> >>> 2018-02-03 1:03 GMT+03:00 Denis Magda <dma...@apache.org>: >>> >>>> Dmitriy, >>>> >>>> I’m surprised to hear that Random-LRU works at the entry level when >> Ignite >>>> persistence is off. Frankly, this section confuses a lot: >>>> https://cwiki.apache.org/confluence/display/IGNITE/ >>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory- >>>> underthehood-Entryeviction <https://cwiki.apache.org/ >>>> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+ >>>> the+hood#IgniteDurableMemory-underthehood-Entryeviction> >>>> >>>> While it was assumed that the entry-based eviction works only for >> on-heap >>>> caches: >>>> https://apacheignite.readme.io/docs/evictions < >>>> https://apacheignite.readme.io/docs/evictions> >>>> >>>> Alex G., please chime in and clarify the misunderstanding. >>>> >>>> — >>>> Denis >>>> >>>>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov <dpavlov....@gmail.com> >> wrote: >>>>> >>>>> Hi Val, >>>>> >>>>> I think it is quite accurate because eviction in case PDS enabled or >> not >>>>> has quite different purposes. >>>>> >>>>> 1. Let us consider PDS enabled and page eviction occurs. First of all >> it >>>> is >>>>> page based eviction, but not entry-based. Actually data is not removed, >>>> but >>>>> only written to disk. We can address this page later by ID. >>>>> PDS eviction is primarily the replacement of pages from memory to page >> on >>>>> disk. This eviction policy should ensure the maximum performance of the >>>>> solution in the future. There is no data removal from grid in this >> case. >>>>> And Ignite does not allow users to configure the policy. If benchmarks >>>> show >>>>> that a change in policy results in increased performance, then we can >>>>> switch policy. >>>>> >>>>> 2. Entry-based eviction (if there is no persistence is available) is >>>>> slithly different. User will need to reload data into grid in case >> entry >>>> is >>>>> evicted but required in cache. In that case Ignite provides selection >> of >>>>> policies. >>>>> >>>>> Sincerely, >>>>> Dmirtiy Pavlov >>>>> >>>>> >>>>> >>>>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko < >>>>> valentin.kuliche...@gmail.com>: >>>>> >>>>>> Guys, >>>>>> >>>>>> Thanks for responses. But I still fail to understand how it affects a >>>> user. >>>>>> >>>>>> Are you saying that with persistence enabled Random-LRU is always used >>>>>> regardless of what is specified in pageEvictionMode, while in >> in-memory >>>>>> only scenario we can also utilize Random-2-LRU for eviction? Is this >>>>>> accurate? Are there any other differences? >>>>>> >>>>>> -Val >>>>>> >>>>>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov <dpavlov....@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Vyacheslav, >>>>>>> >>>>>>> Yes, this is right, but for now PDS page based eviction uses >>>> Random-LRU, >>>>>>> not Random-2-LRU. Something may be changed in future Ignite releases, >>>> but >>>>>>> now Random-LRU is used. I am going to research if there is some room >>>> for >>>>>>> improvements in this aspect (e.g. IGNITE-7507 recenly found). >>>>>>> >>>>>>> If clean page is evicted, nothing really happends. If dirty page is >>>>>> evicted >>>>>>> from region during checkpointing process run, page is stored in file >>>>>> before >>>>>>> eviction >>>>>>> >>>>>>> Some info about this can be found in >>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/ >>>>>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory- >>>>>>> underthehood-Pagebasedeviction >>>>>>> >>>>>>> Sincerely, >>>>>>> Dmitriy Pavlov >>>>>>> >>>>>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur <daradu...@gmail.com >>> : >>>>>>> >>>>>>>> Hi Valentin, >>>>>>>> >>>>>>>> As far as I know, when persistence is enabled and Ignite can't >>>>>>>> allocate new page-memory in the memory segment, then Ignite >>>>>>>> automatically evict some data from RAM to PDS using Random-2-LRU >>>>>>>> eviction algorithm. And there are no mechanisms to evict entries out >>>>>>>> when PDS is enabled. >>>>>>>> >>>>>>>> I'll be glad if I am not right and someone will correct me because, >> as >>>>>>>> Ignite user, I can't use PDS only as RAM disk replication for the >> use >>>>>>>> case in my project. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko >>>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>>> Folks, >>>>>>>>> >>>>>>>>> On "Eviction Policies" documentation page [1] we have the following >>>>>>>> callout: >>>>>>>>> >>>>>>>>>> Configured eviction policy has no effect if Ignite persistence is >>>>>>>> enabled >>>>>>>>>> Note that if Ignite Persistence is enabled, then the page-based >>>>>>>> evictions >>>>>>>>> have no effect because the oldest pages will be purged from memory >>>>>>>>> automatically. >>>>>>>>> >>>>>>>>> This really confuses me. Why is there a difference in how data is >>>>>>> evicted >>>>>>>>> from memory depending on having disk enabled or not? And how does >>>>>>>> eviction >>>>>>>>> work with persistence enabled then? >>>>>>>>> >>>>>>>>> Does anyone can clarify? >>>>>>>>> >>>>>>>>> [1] https://apacheignite.readme.io/docs/evictions >>>>>>>>> >>>>>>>>> -Val >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, Vyacheslav D. >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >> >>
