Hi, Stan!
As a result of personal correspondence I realized that you are right about
making changes:
1)Move warm-up configuration to
org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
2)Start warming up for each region sequentially;
3)Improving warm-up interface:
package org.apache.ignite.internal.processors.cache.warmup;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
/**
* Interface for warming up.
*/
public interface WarmUpStrategy<T extends WarmUpConfiguration> {
/**
* Returns configuration class for mapping to strategy.
*
* @return Configuration class.
*/
Class<T> configClass();
/**
* Warm up.
*
* @param kernalCtx Kernal context.
* @param cfg Warm-up configuration.
* @param region Data region.
* @throws IgniteCheckedException if faild.
*/
void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws
IgniteCheckedException;
/**
* Closing warm up.
*
* @throws IgniteCheckedException if faild.
*/
void close() throws IgniteCheckedException;
}
4)Add a command to "control.sh", to stop current warm-up and cancel all others:
--warm-up stop
5)The "load all" strategy will work as long as there is enough RAM and index
pages will also take priority.
04.08.2020, 13:29, "ткаленко кирилл" <tkalkir...@yandex.ru>:
> Hi, Slava!
>
> Thank you for looking at the offer and making fair comments.
>
> I personally discussed with Anton and Alexey because they are author and
> sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant
> and it can be removed.
> I suggest implementing point 3, since it may be independent of point 1. Also,
> the warm-up will always start after restore phase, without subscribing to
> events.
>
> You are right this should be mentioned in the documentation and javadoc.
>> This means that the user's thread, which starts Ignite via
>> Ignition.start(), will wait for ana additional step - cache warm-up.
>> I think this fact has to be clearly mentioned in our documentation (at
>> Javadocat least) because this step can be time-consuming.
>
> My suggestion for implementation:
> 1)Adding a marker interface
> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache
> warming;
> 2)Set only one configuration via
> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
> 3)Add an internal warm-up interface that will start in [1] after [2];
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
>
> /**
> * Interface for warming up.
> */
> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
> /**
> * Returns configuration class for mapping to strategy.
> *
> * @return Configuration class.
> */
> Class<T> configClass();
>
> /**
> * Warm up.
> *
> * @param kernalCtx Kernal context.
> * @param cfg Warm-up configuration.
> * @throws IgniteCheckedException if faild.
> */
> void warmUp(GridKernalContext kernalCtx, T cfg) throws
> IgniteCheckedException;
> }
>
> 4)Adding an internal plugin extension for add own strategies;
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import java.util.Collection;
> import org.apache.ignite.plugin.Extension;
>
> /**
> * Interface for getting warm-up strategies from plugins.
> */
> public interface WarmUpStrategySupplier extends Extension {
> /**
> * Getting warm-up strategies.
> *
> * @return Warm-up strategies.
> */
> Collection<WarmUpStrategy> strategies();
> }
>
> 5)Add a "Load all" strategy that will load everything to memory as long as
> there is space in it. This strategy is suitable if the persistent storage is
> less than RAM.
>
> Any objections or comments?
>
> [1] -
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
> [2] -
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>
> 27.07.2020, 16:48, "Вячеслав Коптилин" <slava.kopti...@gmail.com>:
>> Hello Kirill,
>>
>> Thanks a lot for driving this activity. If I am not mistaken, this
>> discussion relates to IEP-40.
>>
>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>
>> discovery.
>> This means that the user's thread, which starts Ignite via
>> Ignition.start(), will wait for ana additional step - cache warm-up.
>> I think this fact has to be clearly mentioned in our documentation (at
>> Javadocat least) because this step can be time-consuming.
>>
>>> I suggest adding a new interface:
>>
>> I would change it a bit. First of all, it would be nice to place this
>> interface to a public package and get rid of using GridCacheContext,
>> which is an internal class and it should not leak to the public API in any
>> case.
>> Perhaps, this parameter is not needed at all or we should add some public
>> abstraction instead of internal class.
>>
>> package org.apache.ignite.configuration;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.lang.IgniteFuture;
>>
>> public interface CacheWarmupper {
>> /**
>> * Warmup cache.
>> *
>> * @param cachename Cache name.
>> * @return Future cache warmup.
>> * @throws IgniteCheckedException If failed.
>> */
>> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>> }
>>
>> Thanks,
>> S.
>>
>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tkalkir...@yandex.ru>:
>>
>>> Now, after restarting node, we have only cold caches, which at first
>>> requests to them will gradually load data from disks, which can slow down
>>> first calls to them.
>>> If node has more RAM than data on disk, then they can be loaded at start
>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>> caches.
>>>
>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>> descovery.
>>>
>>> I suggest adding a new interface:
>>>
>>> package org.apache.ignite.internal.processors.cache;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>> import org.jetbrains.annotations.Nullable;
>>>
>>> /**
>>> * Interface for warming up cache.
>>> */
>>> public interface CacheWarmup {
>>> /**
>>> * Warmup cache.
>>> *
>>> * @param cacheCtx Cache context.
>>> * @return Future cache warmup.
>>> * @throws IgniteCheckedException if failed.
>>> */
>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>> throws IgniteCheckedException;
>>> }
>>>
>>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>
>>> Also adding the ability to customize via methods:
>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>
>>> Which will allow for each cache to set implementation of cache warming up,
>>> both for a specific cache, and for all if necessary.
>>>
>>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>>
>>> Questions, suggestions, comments?
>>>
>>> [1] -
>>>
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>> [2] -
>>>
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>> [3] -
>>>
>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
