Hi, AI 2.7.6 doesn't contain a bug with aggregation of cache hits/misses.
I don't sure that described problem is related with IGNITE-3495 [1]. So it makes sense to file an issue. [1] https://issues.apache.org/jira/browse/IGNITE-3495 On Thu, Mar 12, 2020 at 8:21 PM Dominik Przybysz <alien11...@gmail.com> wrote: > > Hi, > I used ignite in version 2.7.6 (but I have also seen this behaviour on other > 2.7.x versions) and there aren't any near or local cache. > I expect that if I ask distributed cache about key which does not exist then > the miss metric will be incremented. > > > śr., 11 mar 2020 o 11:35 Andrey Gura <ag...@apache.org> napisał(a): >> >> Denis, >> >> I don't sure that I understand what is expected behavior should be. >> There are local and aggregated cluster wide metrics. I don't know >> which one used by Visor because I never used it :) >> >> Also it would be great to know what version of Apache Ignite used in >> described case. I remember some bug with metrics aggregation during >> discovery metrics message round trip. >> >> On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dma...@apache.org> wrote: >> > >> > @Nikolay Izhikov <nizhi...@apache.org>, @Andrey Gura <ag...@apache.org>, >> > could you folks check out this thread? >> > >> > I have a feeling that what Dominik is describing was talked out before and >> > rather some sort of a limitation than an issue with the current >> > implementation. >> > >> > - >> > Denis >> > >> > >> > On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <alien11...@gmail.com> >> > wrote: >> > >> > > Hi, >> > > I am trying to use partitioned cache on server nodes to which I connect >> > > with client node. Statistics of cache in the cluster are updated, but >> > > only >> > > for hits metric - misses metric is always 0. >> > > >> > > To reproduce this problem I created cluster of two nodes: >> > > >> > > Server node 1 adds 100 random test cases and prints cache statistics >> > > continuously: >> > > >> > > public class IgniteClusterNode1 { >> > > public static void main(String[] args) throws InterruptedException { >> > > IgniteConfiguration igniteConfiguration = new >> > > IgniteConfiguration(); >> > > >> > > CacheConfiguration cacheConfiguration = new CacheConfiguration(); >> > > cacheConfiguration.setName("test"); >> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); >> > > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); >> > > cacheConfiguration.setStatisticsEnabled(true); >> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration); >> > > >> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi(); >> > > communicationSpi.setLocalPort(47500); >> > > igniteConfiguration.setCommunicationSpi(communicationSpi); >> > > >> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi(); >> > > discoverySpi.setLocalPort(47100); >> > > discoverySpi.setLocalPortRange(100); >> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(); >> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200", >> > > "127.0.0.1:48100..48200")); >> > > igniteConfiguration.setDiscoverySpi(discoverySpi); >> > > >> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) { >> > > try (IgniteCache<String, String> cache = >> > > ignite.getOrCreateCache("test")) { >> > > new Random().ints(1000).map(i -> Math.abs(i % >> > > 1000)).distinct().limit(100).forEach(i -> { >> > > String key = "data_" + i; >> > > String value = UUID.randomUUID().toString(); >> > > cache.put(key, value); >> > > } >> > > ); >> > > } >> > > while (true) { >> > > System.out.println(ignite.cache("test").metrics()); >> > > Thread.sleep(5000); >> > > } >> > > } >> > > } >> > > } >> > > >> > > Server node 2 only prints cache statistics continuously: >> > > >> > > public class IgniteClusterNode2 { >> > > public static void main(String[] args) throws InterruptedException { >> > > IgniteConfiguration igniteConfiguration = new >> > > IgniteConfiguration(); >> > > >> > > CacheConfiguration cacheConfiguration = new CacheConfiguration(); >> > > cacheConfiguration.setName("test"); >> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); >> > > cacheConfiguration.setStatisticsEnabled(true); >> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration); >> > > >> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi(); >> > > communicationSpi.setLocalPort(48500); >> > > igniteConfiguration.setCommunicationSpi(communicationSpi); >> > > >> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi(); >> > > discoverySpi.setLocalPort(48100); >> > > discoverySpi.setLocalPortRange(100); >> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(); >> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200", >> > > "127.0.0.1:48100..48200")); >> > > igniteConfiguration.setDiscoverySpi(discoverySpi); >> > > >> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) { >> > > while (true) { >> > > System.out.println(ignite.cache("test").metrics()); >> > > Thread.sleep(5000); >> > > } >> > > } >> > > } >> > > } >> > > >> > > Next I start a client node which continuously read data from the cluster: >> > > >> > > public class CacheClusterReader { >> > > public static void main(String[] args) throws InterruptedException { >> > > IgniteConfiguration cfg = new IgniteConfiguration(); >> > > cfg.setClientMode(true); >> > > >> > > TcpDiscoverySpi spi = new TcpDiscoverySpi(); >> > > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder(); >> > > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200", >> > > "127.0.0.1:48100..48200")); >> > > spi.setIpFinder(tcMp); >> > > cfg.setDiscoverySpi(spi); >> > > >> > > CacheConfiguration<String, String> cacheConfig = new >> > > CacheConfiguration<>("test"); >> > > cacheConfig.setStatisticsEnabled(true); >> > > cacheConfig.setCacheMode(CacheMode.PARTITIONED); >> > > cfg.setCacheConfiguration(cacheConfig); >> > > >> > > try (Ignite ignite = Ignition.start(cfg)) { >> > > System.out.println(ignite.cacheNames()); >> > > >> > > while (true) { >> > > try (IgniteCache<String, String> cache = >> > > ignite.getOrCreateCache(cacheConfig)) { >> > > cache.enableStatistics(true); >> > > >> > > new Random().ints(1000).map(i -> Math.abs(i % >> > > 2000)).distinct().limit(100).forEach(i -> { >> > > String key = "data_" + i; >> > > String value = cache.get(key); >> > > System.out.println("Found " + key + " >> > > with >> > > value " + value); >> > > } >> > > ); >> > > >> > > System.out.println(cache.metrics()); >> > > System.out.println(cache.localMetrics()); >> > > } >> > > Thread.sleep(3000); >> > > } >> > > } >> > > } >> > > } >> > > >> > > On server and client nodes cluster statistics are synchronized, but >> > > misses >> > > counter is always equal to 0, e. g. >> > > >> > > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0, >> > > entryProcessorReadOnlyInvocations=0, >> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0, >> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0, >> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0, >> > > entryProcessorMaxInvocationTime=0.0, >> > > entryProcessorMinInvocationTime=0.0, *hits=93, >> > > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0, >> > > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0, >> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test, >> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0, >> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0, >> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0, >> > > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false, >> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0, >> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0, >> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0, >> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0, >> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0, >> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1, >> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1, >> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1, >> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1, >> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1, >> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, >> > > rebalancingPartitionsCnt=0, >> > > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0, >> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0, >> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0, >> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true, >> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false, >> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true] >> > > >> > > In comparision, I also have also a thin client which connects to server >> > > and read some random data once: >> > > >> > > public class CacheClusterThinClientReader { >> > > public static void main(String[] args) throws Exception { >> > > ClientConfiguration clientConfiguration = new >> > > ClientConfiguration() >> > > .setAddresses("127.0.0.1:10800"); >> > > >> > > try (IgniteClient ignite = >> > > Ignition.startClient(clientConfiguration)) { >> > > >> > > ClientCache<String, String> cache = >> > > ignite.getOrCreateCache("test"); >> > > >> > > Set<String> misses = new HashSet<>(); >> > > new Random().ints(1000).map(i -> Math.abs(i % >> > > 2000)).distinct().limit(100).forEach(i -> { >> > > String key = "data_" + i; >> > > String value = cache.get(key); >> > > System.out.println("Found " + key + " with value >> > > " >> > > + value); >> > > if (value == null) { >> > > misses.add(key); >> > > } >> > > } >> > > ); >> > > System.out.println("Misses count: " + misses.size()); >> > > } >> > > } >> > > } >> > > >> > > and finally prints (in my try): >> > > >> > > Misses count: 97 >> > > >> > > On the server nodes misses were counted, but only for one server node: >> > > >> > > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0, >> > > entryProcessorReadOnlyInvocations=0, >> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0, >> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0, >> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0, >> > > entryProcessorMaxInvocationTime=0.0, >> > > entryProcessorMinInvocationTime=0.0, *hits=3, >> > > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0, >> > > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, >> > > rmvAvgTimeNanos=0.0, >> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test, >> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0, >> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0, >> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0, >> > > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false, >> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0, >> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0, >> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0, >> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0, >> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0, >> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1, >> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1, >> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1, >> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1, >> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1, >> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, >> > > rebalancingPartitionsCnt=0, >> > > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0, >> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0, >> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0, >> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true, >> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false, >> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true] >> > > >> > > Visor confirms my observation: >> > > >> > > Nodes for: test(@c0) >> > > >> > > +==================================================================================================================+ >> > > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time >> > > | Size (Primary / Backup) | Hi/Mi/Rd/Wr | >> > > >> > > +==================================================================================================================+ >> > > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291 >> > > | Total: 47 (47 / 0) | Hi: 0 | >> > > | | | | | >> > > | Heap: 0 (0 / <n/a>) | Mi: 0 | >> > > | | | | | >> > > | Off-Heap: 47 (47 / 0) | Rd: 0 | >> > > | | | | | >> > > | Off-Heap Memory: <n/a> | Wr: 0 | >> > > >> > > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+ >> > > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122 >> > > | Total: 53 (53 / 0) | Hi: 3 | >> > > | | | | | >> > > | Heap: 0 (0 / <n/a>) | Mi: 48 | >> > > | | | | | >> > > | Off-Heap: 53 (53 / 0) | Rd: 51 | >> > > | | | | | >> > > | Off-Heap Memory: <n/a> | Wr: 100 | >> > > >> > > +------------------------------------------------------------------------------------------------------------------+ >> > > >> > > I have seen the same behaviour with rest api as with the thin client, but >> > > I don't have prepared test case for this. >> > > >> > > Is there a good way to correctly configure collecting statistics on >> > > server >> > > nodes? >> > > Should I create an issue for this problem? >> > > >> > > -- >> > > Pozdrawiam / Regards, >> > > Dominik Przybysz >> > > > > > > -- > Pozdrawiam / Regards, > Dominik Przybysz
