Denis,
I don't sure that I understand what is expected behavior should be.
There are local and aggregated cluster wide metrics. I don't know
which one used by Visor because I never used it :)
Also it would be great to know what version of Apache Ignite used in
described case. I remember some bug with metrics aggregation during
discovery metrics message round trip.
On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dma...@apache.org> wrote:
>
> @Nikolay Izhikov <nizhi...@apache.org>, @Andrey Gura <ag...@apache.org>,
> could you folks check out this thread?
>
> I have a feeling that what Dominik is describing was talked out before and
> rather some sort of a limitation than an issue with the current
> implementation.
>
> -
> Denis
>
>
> On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <alien11...@gmail.com>
> wrote:
>
> > Hi,
> > I am trying to use partitioned cache on server nodes to which I connect
> > with client node. Statistics of cache in the cluster are updated, but only
> > for hits metric - misses metric is always 0.
> >
> > To reproduce this problem I created cluster of two nodes:
> >
> > Server node 1 adds 100 random test cases and prints cache statistics
> > continuously:
> >
> > public class IgniteClusterNode1 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(47500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(47100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache("test")) {
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 1000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = UUID.randomUUID().toString();
> > cache.put(key, value);
> > }
> > );
> > }
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Server node 2 only prints cache statistics continuously:
> >
> > public class IgniteClusterNode2 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(48500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(48100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Next I start a client node which continuously read data from the cluster:
> >
> > public class CacheClusterReader {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setClientMode(true);
> >
> > TcpDiscoverySpi spi = new TcpDiscoverySpi();
> > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > spi.setIpFinder(tcMp);
> > cfg.setDiscoverySpi(spi);
> >
> > CacheConfiguration<String, String> cacheConfig = new
> > CacheConfiguration<>("test");
> > cacheConfig.setStatisticsEnabled(true);
> > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > cfg.setCacheConfiguration(cacheConfig);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > System.out.println(ignite.cacheNames());
> >
> > while (true) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache(cacheConfig)) {
> > cache.enableStatistics(true);
> >
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with
> > value " + value);
> > }
> > );
> >
> > System.out.println(cache.metrics());
> > System.out.println(cache.localMetrics());
> > }
> > Thread.sleep(3000);
> > }
> > }
> > }
> > }
> >
> > On server and client nodes cluster statistics are synchronized, but misses
> > counter is always equal to 0, e. g.
> >
> > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0,
> > *hits=93,
> > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > In comparision, I also have also a thin client which connects to server
> > and read some random data once:
> >
> > public class CacheClusterThinClientReader {
> > public static void main(String[] args) throws Exception {
> > ClientConfiguration clientConfiguration = new ClientConfiguration()
> > .setAddresses("127.0.0.1:10800");
> >
> > try (IgniteClient ignite =
> > Ignition.startClient(clientConfiguration)) {
> >
> > ClientCache<String, String> cache =
> > ignite.getOrCreateCache("test");
> >
> > Set<String> misses = new HashSet<>();
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with value "
> > + value);
> > if (value == null) {
> > misses.add(key);
> > }
> > }
> > );
> > System.out.println("Misses count: " + misses.size());
> > }
> > }
> > }
> >
> > and finally prints (in my try):
> >
> > Misses count: 97
> >
> > On the server nodes misses were counted, but only for one server node:
> >
> > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0,
> > *hits=3,
> > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > Visor confirms my observation:
> >
> > Nodes for: test(@c0)
> >
> > +==================================================================================================================+
> > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
> >
> > +==================================================================================================================+
> > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
> > | Total: 47 (47 / 0) | Hi: 0 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 0 |
> > | | | | |
> > | Off-Heap: 47 (47 / 0) | Rd: 0 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 0 |
> >
> > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
> > | Total: 53 (53 / 0) | Hi: 3 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 48 |
> > | | | | |
> > | Off-Heap: 53 (53 / 0) | Rd: 51 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 100 |
> >
> > +------------------------------------------------------------------------------------------------------------------+
> >
> > I have seen the same behaviour with rest api as with the thin client, but
> > I don't have prepared test case for this.
> >
> > Is there a good way to correctly configure collecting statistics on server
> > nodes?
> > Should I create an issue for this problem?
> >
> > --
> > Pozdrawiam / Regards,
> > Dominik Przybysz
> >
