Hi,
I am prototyping using ignite to ingest lots of short lived events in a
system. In order to ingest the data i'm using the kafka streamer mechanism.
I'm pushing 20k events / second into the system into a partitioned off heap
cache. I've run into a few performance issues but I've been able to wiggle
my way out of it using information from the forums and ignite docs. But the
latest issue I can't seem to find any information about.
My cache is setup to evict the data after one hour. The insertion
performance degrades substantially when the eviction mechanism kicks in
where the latency of the insertion continues to degrade over time.
WRT to the OS, memory and disk none of them are saturated. The load avg is
low, the gc is not under pressure and the disk is not fully utilized.
I've experimented around with different eviction times to make sure that it
is indeed the eviction that is causing this. After some idle time I startup
our event simulator and wait for the evictions to start and sure enough the
latency of insertions increases immediately when evictions start.
Each node is 32 GB / 8 cpus.
Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
-XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
-XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
-XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
-XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
-XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
I was wondering if there is no tuning I can do to avoid this overhead, I am
partitioning my data by time - is there a performant way to clear a
partition in a cache one at a time? (i was thinking of something similar to
a truncate command in a rdbms)
Here is my schema:
CachePojo {
@QuerySqlField(index=false)
private String buffer0;
@QuerySqlField(index=true)
private String buffer1;
@QuerySqlField
private String buffer2;
@QuerySqlField(index=true, inlineSize=8, descending=true)
private Timestamp timestamp;
@QuerySqlField(index=false)
private Map<String, String> fields = new HashMap<>();
}
Ignite config:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setIgniteHome(igniteHome);
// this is probably not needed
igniteConfiguration.setTransactionConfiguration(new
TransactionConfiguration());
igniteConfiguration.setPeerClassLoadingEnabled(true);
igniteConfiguration.setIncludeEventTypes(new int[0]);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
igniteConfiguration.setDataStreamerThreadPoolSize(8);
igniteConfiguration.setPublicThreadPoolSize(16);
igniteConfiguration.setSystemThreadPoolSize(16);
DataStorageConfiguration dsc = dataStorageConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsc);
dataStorageConfig:
DataStorageConfiguration dataStorageConfiguration = new
DataStorageConfiguration();
DataRegionConfiguration dataRegionConfiguration =
dataRegionConfiguration();
dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
dataStorageConfiguration.setDataRegionConfigurations(largeDataRegionConfiguration());
dataStorageConfiguration.setPageSize(8192);
dataStorageConfiguration.setMetricsEnabled(true);
dataStorageConfiguration.setWriteThrottlingEnabled(true);
dataStorageConfiguration.setStoragePath("/var/lib/ignite/persistence");
dataStorageConfiguration.setWalMode(WALMode.NONE);
dataStorageConfiguration.setWalPath("/var/lib/ignite/wal");
dataStorageConfiguration.setWalArchivePath("/var/lib/ignite/wal/archive");
cacheDataRegionConfig:
DataRegionConfiguration dataRegionConfiguration = new
DataRegionConfiguration();
dataRegionConfiguration.setPersistenceEnabled(true);
dataRegionConfiguration.setName("dataRegion");
dataRegionConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
dataRegionConfiguration.setInitialSize(500l * 1024 * 1024);
dataRegionConfiguration.setMaxSize(8l * 1024 * 1024 * 1024);
dataRegionConfiguration.setMetricsEnabled(true);
dataRegionConfiguration.setEmptyPagesPoolSize(1024);
cacheConfig:
CacheConfiguration<Key, CachePojo> cacheConfig = new
CacheConfiguration<>();
cacheConfig.setSqlFunctionClasses(SearchFunctions.class);
cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 60)));
cacheConfig.setName("mycache");
cacheConfig.setTypes(Key.class, CachePojo.class);
cacheConfig.setIndexedTypes(Key.class, CachePojo.class);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfig.setQueryParallelism(64);
cacheConfig.setStatisticsEnabled(true);
cacheConfig.setGroupName("cache");
cacheConfig.setDataRegionName("dataRegion");
// partitions in one minute chunks evenly across all nodes
cacheConfig.setAffinity(new CacheAffinityFunction());
cacheConfig.setRebalanceBatchSize(1024 * 1024 * 4);
cacheConfig.setRebalanceThreadPoolSize(4);
cacheConfig.setRebalanceTimeout(TimeUnit.MINUTES.toMillis(10));
cacheConfig.setRebalanceDelay(TimeUnit.MINUTES.toMillis(2));
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
