Igniters, Seems 2.2 is a urgent bugfix release, so it should be based on 2.1, In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
Currently, I see 835 issues with fixVersion = 2.2 Seems we should have only 4 issues with fixVersion = 2.2: - Change default max memory size from 80% to 20% - https://issues.apache.org/jira/browse/IGNITE-6182 - Detecting low memory on ignite node startup - https://issues.apache.org/jira/browse/IGNITE-6003 - Backport of improvements to checkpoint algorithm - https://issues.apache.org/jira/browse/IGNITE-???? - ML profile is missing in 2.1 binary release - https://issues.apache.org/jira/browse/IGNITE-6193 Please correct me in case I've missed something. Ivan, > Let's include to the release tickets with optimizations of checkpointing algorithm: > https://issues.apache.org/jira/browse/IGNITE-6178 > https://issues.apache.org/jira/browse/IGNITE-6033 > https://issues.apache.org/jira/browse/IGNITE-5961 > This will help users who are experiencing problems with slow checkpoints. > Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to soften "Ignite node crashed in the middle of checkpoint" message as per discussion < http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html >. All of these issues should have fixVersion 2.3. Please create backport issue and link it to all necessary issues. P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1. On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <avinogra...@gridgain.com> wrote: > Denis, > > > BTW, who is considered to be the release manager of this release? > I'll do it. > > On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gvvinbl...@gmail.com> > wrote: > >> Ok, the check happens at the node start time or on NODE_JOIN event >> >> in general it looks like: >> >> 1) calculate expected used memory = heap max + system cache max + all >> custom policies max + default policy size and put it into a node attribute >> >> 2) get total physycal memory, calculate expected safe to be used memory >> amount (leave 4 gb min or 20% of available memory for OS) >> >> 3) if expected used memory + expected used memory of other nodes on the >> host > than safe to be used memory amount, start calculating suggestions >> >> 4) Each ignite instance needs at least 512mb heap + 40mb system cache + >> 100mb default polycy, if available memory is less we cannot suggest >> anything reasonable, print warning, stop calculation. >> >> 5) check heap size (shouldn't exceed 30% of available memory (total_memory >> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just >> calculated value or 512MB minimal) >> >> 6) check if system cache size changed, suggest default value if it's so >> >> 7) in case 100 mb * policies count < available memory, suggest using >> default policy with max size equals to remaining memory (available - heap >> - >> system cache) >> >> 8) calculate new size for each memory policy ( it's user defined size * >> (remaining / (all_policies_size * nodes_cnt)); in proportion to >> remaining memory, devided by nodes number on the host or 100 mb minimal) >> >> 9) print suggestions >> >> >> >> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: >> >> > Igor, can you please describe the algorithm with all the thresholds? >> > >> > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinbl...@gmail.com >> > >> > wrote: >> > >> > > The suggestion here is based on initial settings, and it's so because >> > there >> > > is no other nodes on the host in the example. >> > > >> > > The algorithm tries to preserve the original ratio of memory policies >> > > keeping numbers reasonable (for example after some thresshold it will >> > > suggest not to use several memory policies if there is not enough of >> > memory >> > > for all of them) and taking into consideration nodes count on the >> host, >> > > each jvm heap, needed memory for OS, etc >> > > >> > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: >> > > >> > > > Looks good, but why in the example provided are we suggesting 8GB? 2 >> > > nodes >> > > > with 8GB will completely exhaust the available memory. I would >> suggest >> > 6 >> > > or >> > > > 7GB. >> > > > >> > > > Also, why 100MB for default policy. Anything under 1GB seems too >> small. >> > > > >> > > > Can you please comment? >> > > > >> > > > D. >> > > > >> > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor < >> > gvvinbl...@gmail.com> >> > > > wrote: >> > > > >> > > > > One more example of possible warning: >> > > > > >> > > > > ----------------------------------------------------- >> > > > > Excessive memory usage by Ignite node process (performance may >> drop) >> > > > > [requested=44613MB, available=15942MB]. >> > > > > >> > > > > Please tune the folowing settings as suggested: >> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB >> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB >> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB >> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB >> > > > > >> > > > > Current settings: >> > > > > Java Heap maxSize: 3543MB >> > > > > Java Heap initSize: 250MB >> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB >> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB >> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB >> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB >> > > > > The overall expected memory usage by all Ignite nodes on the >> host: >> > > > > 44613MB >> > > > > ----------------------------------------------------- >> > > > > >> > > > > Your thoughts? >> > > > > >> > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dma...@apache.org>: >> > > > > >> > > > > > Guys, >> > > > > > >> > > > > > ML lib profile is missing in 2.1 release! That must be fixed and >> > > rolled >> > > > > > out in this emergency release: >> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193 < >> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193> >> > > > > > >> > > > > > Oleg, Yuri, please step in and handle the issue. >> > > > > > >> > > > > > BTW, who is considered to be the release manager of this >> release? >> > > > > > >> > > > > > — >> > > > > > Denis >> > > > > > >> > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan < >> > > > dsetrak...@apache.org> >> > > > > > wrote: >> > > > > > > >> > > > > > > I like the format proposed by Denis, very clear. >> > > > > > > >> > > > > > > However, I also do not understand why a user should change the >> > size >> > > > of >> > > > > > some >> > > > > > > system cache. How would a user ever know what value to put >> there? >> > > > This >> > > > > > > value should be configured by Ignite automatically. >> > > > > > > >> > > > > > > D. >> > > > > > > >> > > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda < >> dma...@apache.org> >> > > > > wrote: >> > > > > > > >> > > > > > >> Igor, >> > > > > > >> >> > > > > > >> Let me suggest this format. >> > > > > > >> >> > > > > > >> --------------------------------------------- >> > > > > > >> Excessive memory usage by Ignite node process (performance >> may >> > > drop) >> > > > > > >> [requested=29251MB, available=15942MB] >> > > > > > >> >> > > > > > >> Please tune the following settings: >> > > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested >> > value] >> > > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value >> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = >> > > suggested >> > > > > > >> value] >> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = >> > > suggested >> > > > > > >> value] >> > > > > > >> >> > > > > > >> Current settings: >> > > > > > >> [DefaultMemoryPolicySize = value] >> > > > > > >> [{policy_name_1} size = value] >> > > > > > >> [{policy_name_1} size = value] >> > > > > > >> SystemCacheInitialSize = value >> > > > > > >> SystemCacheMaxSize = value >> > > > > > >> Java Heap Init Size = value >> > > > > > >> Java Heap Max Size = value >> > > > > > >> >> > > > > > >> The overall memory usage by all Ignite nodes on the host: >> value >> > > > > > >> ------------------------------------------- >> > > > > > >> >> > > > > > >> Records in […] are optional. If custom memory policy is not >> set >> > or >> > > > the >> > > > > > >> default memory policy is overridden the output will miss >> some of >> > > the >> > > > > > rows. >> > > > > > >> >> > > > > > >> As for systemCacheMaxSize, it should be show ONLY if the >> > parameter >> > > > was >> > > > > > set >> > > > > > >> explicitly by user code. Otherwise, the platform should be >> wise >> > > > enough >> > > > > > to >> > > > > > >> instantiate it properly depending on the host memory usage. >> > > > > > >> >> > > > > > >> — >> > > > > > >> Denis >> > > > > > >> >> > > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor < >> > > > gvvinbl...@gmail.com> >> > > > > > >> wrote: >> > > > > > >>> >> > > > > > >>> The message without logging layout: >> > > > > > >>> >> > > > > > >>> Not enough memory for current process [required=29251MB, >> > > > > > >> available=15942MB]. >> > > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and >> > > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease >> memory >> > > > > > allocated >> > > > > > >>> for each node. >> > > > > > >>> >> > > > > > >>> Current settings: >> > > > > > >>> HeapInit=250MB >> > > > > > >>> HeapMax=3543MB >> > > > > > >>> DefaultMemoryPolicySize=12753MB >> > > > > > >>> SystemCacheInitialSize=40MB >> > > > > > >>> SystemCacheMaxSize=100MB >> > > > > > >>> >> > > > > > >>> Other ignite instanses on the server require: 12853MB >> > > > > > >>> >> > > > > > >>> I think it's make sense to describe what these numbers are >> > > consist >> > > > > of. >> > > > > > >>> >> > > > > > >>> We simple say which parameters have an impact on how much >> > memory >> > > > the >> > > > > > >>> instance needs and their (parameters) actual values. >> > > > > > >>> >> > > > > > >>> Also we notice that more than one Ignite instance are ran on >> > the >> > > > > server >> > > > > > >> or >> > > > > > >>> workstation and it also consumes memory. >> > > > > > >>> >> > > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" < >> > > > > > >>> dsetrak...@apache.org> написал: >> > > > > > >>> >> > > > > > >>>> Igor, what is this flood of WARN messaging coming after the >> > > text? >> > > > > Are >> > > > > > we >> > > > > > >>>> really going to print this whole thing out? >> > > > > > >>>> >> > > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor < >> > > > > > gvvinbl...@gmail.com >> > > > > > >>> >> > > > > > >>>> wrote: >> > > > > > >>>> >> > > > > > >>>>> This message appears on topology change in case the >> available >> > > > > memory >> > > > > > is >> > > > > > >>>>> exceeded >> > > > > > >>>>> >> > > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor < >> > > > gvvinbl...@gmail.com >> > > > > >: >> > > > > > >>>>> >> > > > > > >>>>>> An example of current impl: >> > > > > > >>>>>> >> > > > > > >>>>>> >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough >> > > memory >> > > > > for >> > > > > > >>>>>> current process [required=29251MB, available=15942MB]. >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please >> change >> > > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and >> > > MemoryConfiguration. >> > > > > > >>>>> defaultMemoryPolicySize >> > > > > > >>>>>> to decrease memory allocated for each node. >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current >> > > settings: >> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > HeapInit=250MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > HeapMax=3543MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>> DefaultMemoryPolicySize= >> > > > > > >>>>>> 12753MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>>>> SystemCacheInitialSize=40MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>> SystemCacheMaxSize=100MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other >> ignite >> > > > > > instanses >> > > > > > >>>> on >> > > > > > >>>>>> the server require: 12853MB >> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% >> > > > internal. >> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] >> > > > > > >>>>>> >> > > > > > >>>>>> >> > > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov < >> > > skoz...@gridgain.com >> > > > >: >> > > > > > >>>>>> >> > > > > > >>>>>>> I suppose we should not forget JVM heap size and >> suggest to >> > > > > reduce >> > > > > > >>>> both >> > > > > > >>>>>>> options >> > > > > > >>>>>>> >> > > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan < >> > > > > > >>>>> dsetrak...@apache.org >> > > > > > >>>>>>>> >> > > > > > >>>>>>> wrote: >> > > > > > >>>>>>> >> > > > > > >>>>>>>> Igor, I would change the message. How about this: >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> Required RAM size is larger than total physical memory >> > > > available >> > > > > > for >> > > > > > >>>>> OS. >> > > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and >> > > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease >> > memory >> > > > > > >>>>> allocated >> > > > > > >>>>>>> for >> > > > > > >>>>>>>>> each node. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> Also, can we calculate what the memory size allocated >> for >> > > each >> > > > > > node >> > > > > > >>>>>>> should >> > > > > > >>>>>>>> be? In that case we should suggest it. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> D. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor < >> > > > > > >>>>> gvvinbl...@gmail.com >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> wrote: >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> What do you, guys think about next warning? >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO >> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ >> > > GridHomePat >> > > > > > >>>>>>> hSelfTest0] >> > > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 >> MB. >> > > Use >> > > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to >> > > change >> > > > > the >> > > > > > >>>>>>>> setting. >> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN >> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ >> > > GridHomePat >> > > > > > >>>>>>> hSelfTest0] >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>>>>> Required RAM size is larger than total physical >> memory >> > > > > > >>>> available >> > > > > > >>>>>>> for >> > > > > > >>>>>>>>> OS. >> > > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition >> > usage. >> > > > > > >>>>>>>>>>>> Use MemoryConfiguration and >> MemoryPolicyConfiguration >> > to >> > > > > > >>>> change >> > > > > > >>>>>>> the >> > > > > > >>>>>>>>> settings. >> > > > > > >>>>>>>>>>>> Physical memory [required=16397MB, >> available=15942MB] >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN >> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ >> > > GridHomePat >> > > > > > >>>>>>> hSelfTest0] >> > > > > > >>>>>>>>> Peer class loading is enabled (disable it in >> production >> > for >> > > > > > >>>>>>> performance >> > > > > > >>>>>>>> and >> > > > > > >>>>>>>>> deployment consistency reasons) >> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO >> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ >> > > GridHomePat >> > > > > > >>>>>>> hSelfTest0] >> > > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy: >> > > > > > >>>>> ['ignite-sys-cache']] >> > > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO >> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ >> > > GridHomePat >> > > > > > >>>>>>> hSelfTest0] >> > > > > > >>>>>>>>> 3-rd party licenses can be found at: >> > > > > > >>>>>>>>> /home/gvvinblade/projects/igni >> te/incubator-ignite/libs/ >> > > > > licenses >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov < >> > > > yzhda...@apache.org >> > > > > >: >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>>> Agree, let's release new version including tickets >> > > mentioned >> > > > > by >> > > > > > >>>>>>> Denis >> > > > > > >>>>>>>> and >> > > > > > >>>>>>>>>> Ivan. >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>>> --Yakov >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> -- >> > > > > > >>>>>>> Sergey Kozlov >> > > > > > >>>>>>> GridGain Systems >> > > > > > >>>>>>> www.gridgain.com >> > > > > > >>>>>>> >> > > > > > >>>>>> >> > > > > > >>>>>> >> > > > > > >>>>> >> > > > > > >>>> >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
