Great! Then I proceed to ticket
https://issues.apache.org/jira/browse/IGNITE-13345.
10.08.2020, 16:30, "Stanislav Lukyanov" <stanlukya...@gmail.com>:
> All of this looks awesome, covers the use cases I know about.
> Thanks!
>
> Stan
>
>> On 10 Aug 2020, at 15:39, ткаленко кирилл <tkalkir...@yandex.ru> wrote:
>>
>> Hi, Stan again :-)
>>
>> I suggest adding a little more flexibility to configuration:
>> 1)Add default warm-up configuration for all regions into
>> org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
>> 2)Add a NoOp strategy for turning off heating of a specific region
>>
>> Thus, when starting warm-up, region configuration is taken at beginning,
>> and if it is not present, it is taken from default. And if we don't want to
>> warm up region, we set NoOp.
>>
>> 10.08.2020, 10:20, "ткаленко кирилл" <tkalkir...@yandex.ru>:
>>> Hi, Stan!
>>>
>>> As a result of personal correspondence I realized that you are right about
>>> making changes:
>>> 1)Move warm-up configuration to
>>> org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>>> 2)Start warming up for each region sequentially;
>>> 3)Improving warm-up interface:
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>> import org.apache.ignite.internal.GridKernalContext;
>>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>>
>>> /**
>>> * Interface for warming up.
>>> */
>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>> /**
>>> * Returns configuration class for mapping to strategy.
>>> *
>>> * @return Configuration class.
>>> */
>>> Class<T> configClass();
>>>
>>> /**
>>> * Warm up.
>>> *
>>> * @param kernalCtx Kernal context.
>>> * @param cfg Warm-up configuration.
>>> * @param region Data region.
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region)
>>> throws IgniteCheckedException;
>>>
>>> /**
>>> * Closing warm up.
>>> *
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void close() throws IgniteCheckedException;
>>> }
>>>
>>> 4)Add a command to "control.sh", to stop current warm-up and cancel all
>>> others: --warm-up stop
>>> 5)The "load all" strategy will work as long as there is enough RAM and
>>> index pages will also take priority.
>>>
>>> 04.08.2020, 13:29, "ткаленко кирилл" <tkalkir...@yandex.ru>:
>>>> Hi, Slava!
>>>>
>>>> Thank you for looking at the offer and making fair comments.
>>>>
>>>> I personally discussed with Anton and Alexey because they are author and
>>>> sponsor of "IEP-40" and we found out that point 2 in it is no longer
>>>> relevant and it can be removed.
>>>> I suggest implementing point 3, since it may be independent of point 1.
>>>> Also, the warm-up will always start after restore phase, without
>>>> subscribing to events.
>>>>
>>>> You are right this should be mentioned in the documentation and javadoc.
>>>>> This means that the user's thread, which starts Ignite via
>>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>>> Javadocat least) because this step can be time-consuming.
>>>>
>>>> My suggestion for implementation:
>>>> 1)Adding a marker interface
>>>> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring
>>>> cache warming;
>>>> 2)Set only one configuration via
>>>> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>>> 3)Add an internal warm-up interface that will start in [1] after [2];
>>>>
>>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>> import org.apache.ignite.IgniteCheckedException;
>>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>>> import org.apache.ignite.internal.GridKernalContext;
>>>>
>>>> /**
>>>> * Interface for warming up.
>>>> */
>>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>>> /**
>>>> * Returns configuration class for mapping to strategy.
>>>> *
>>>> * @return Configuration class.
>>>> */
>>>> Class<T> configClass();
>>>>
>>>> /**
>>>> * Warm up.
>>>> *
>>>> * @param kernalCtx Kernal context.
>>>> * @param cfg Warm-up configuration.
>>>> * @throws IgniteCheckedException if faild.
>>>> */
>>>> void warmUp(GridKernalContext kernalCtx, T cfg) throws
>>>> IgniteCheckedException;
>>>> }
>>>>
>>>> 4)Adding an internal plugin extension for add own strategies;
>>>>
>>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>> import java.util.Collection;
>>>> import org.apache.ignite.plugin.Extension;
>>>>
>>>> /**
>>>> * Interface for getting warm-up strategies from plugins.
>>>> */
>>>> public interface WarmUpStrategySupplier extends Extension {
>>>> /**
>>>> * Getting warm-up strategies.
>>>> *
>>>> * @return Warm-up strategies.
>>>> */
>>>> Collection<WarmUpStrategy> strategies();
>>>> }
>>>>
>>>> 5)Add a "Load all" strategy that will load everything to memory as long
>>>> as there is space in it. This strategy is suitable if the persistent
>>>> storage is less than RAM.
>>>>
>>>> Any objections or comments?
>>>>
>>>> [1] -
>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>> [2] -
>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>
>>>> 27.07.2020, 16:48, "Вячеслав Коптилин" <slava.kopti...@gmail.com>:
>>>>> Hello Kirill,
>>>>>
>>>>> Thanks a lot for driving this activity. If I am not mistaken, this
>>>>> discussion relates to IEP-40.
>>>>>
>>>>>> I suggest adding a warmup phase after recovery here [1] after [2],
>>>>>> before
>>>>>
>>>>> discovery.
>>>>> This means that the user's thread, which starts Ignite via
>>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>>> Javadocat least) because this step can be time-consuming.
>>>>>
>>>>>> I suggest adding a new interface:
>>>>>
>>>>> I would change it a bit. First of all, it would be nice to place this
>>>>> interface to a public package and get rid of using GridCacheContext,
>>>>> which is an internal class and it should not leak to the public API in
>>>>> any
>>>>> case.
>>>>> Perhaps, this parameter is not needed at all or we should add some
>>>>> public
>>>>> abstraction instead of internal class.
>>>>>
>>>>> package org.apache.ignite.configuration;
>>>>>
>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>> import org.apache.ignite.lang.IgniteFuture;
>>>>>
>>>>> public interface CacheWarmupper {
>>>>> /**
>>>>> * Warmup cache.
>>>>> *
>>>>> * @param cachename Cache name.
>>>>> * @return Future cache warmup.
>>>>> * @throws IgniteCheckedException If failed.
>>>>> */
>>>>> IgniteFuture<?> warmup(String cachename) throws
>>>>> IgniteCheckedException;
>>>>> }
>>>>>
>>>>> Thanks,
>>>>> S.
>>>>>
>>>>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tkalkir...@yandex.ru>:
>>>>>
>>>>>> Now, after restarting node, we have only cold caches, which at first
>>>>>> requests to them will gradually load data from disks, which can slow
>>>>>> down
>>>>>> first calls to them.
>>>>>> If node has more RAM than data on disk, then they can be loaded at
>>>>>> start
>>>>>> "warmup", thereby solving the issue of slowdowns during first calls
>>>>>> to
>>>>>> caches.
>>>>>>
>>>>>> I suggest adding a warmup phase after recovery here [1] after [2],
>>>>>> before
>>>>>> descovery.
>>>>>>
>>>>>> I suggest adding a new interface:
>>>>>>
>>>>>> package org.apache.ignite.internal.processors.cache;
>>>>>>
>>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>>>>> import org.jetbrains.annotations.Nullable;
>>>>>>
>>>>>> /**
>>>>>> * Interface for warming up cache.
>>>>>> */
>>>>>> public interface CacheWarmup {
>>>>>> /**
>>>>>> * Warmup cache.
>>>>>> *
>>>>>> * @param cacheCtx Cache context.
>>>>>> * @return Future cache warmup.
>>>>>> * @throws IgniteCheckedException if failed.
>>>>>> */
>>>>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext
>>>>>> cacheCtx)
>>>>>> throws IgniteCheckedException;
>>>>>> }
>>>>>>
>>>>>> Which will allow to warm up caches in parallel and asynchronously.
>>>>>> Warmup
>>>>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>>>>
>>>>>> Also adding the ability to customize via methods:
>>>>>>
>>>>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>>>
>>>>>> Which will allow for each cache to set implementation of cache
>>>>>> warming up,
>>>>>> both for a specific cache, and for all if necessary.
>>>>>>
>>>>>> I suggest adding an implementation of SequentialWarmup that will use
>>>>>> [3].
>>>>>>
>>>>>> Questions, suggestions, comments?
>>>>>>
>>>>>> [1] -
>>>>>>
>>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>>> [2] -
>>>>>>
>>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>>> [3] -
>>>>>>
>>>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
