Dear all, I was wondering if this is a know issue which has a chance to be fixed in future or (I hope) it is me who missed something obvious in working with Ignite caches. I have a simple single node test app (built to investigate a memory leak observed in our PROD deployment), that creates c.a. 20 LOCAL caches per sec. with the config below:
private IgniteCache<String, BinaryObject> createLocalCache(String name) {
CacheConfiguration<String, BinaryObject> cCfg = new
CacheConfiguration<>();
cCfg.setName(name);
cCfg.setGroupName("localCaches"); // without group leak is much
bigger!
cCfg.setStoreKeepBinary(true);
cCfg.setCacheMode(CacheMode.LOCAL);
cCfg.setOnheapCacheEnabled(false);
cCfg.setCopyOnRead(false);
cCfg.setBackups(0);
cCfg.setWriteBehindEnabled(false);
cCfg.setReadThrough(false);
cCfg.setReadFromBackup(false);
cCfg.setQueryEntities(<some basic query here>);
return ignite.createCache(cCfg).withKeepBinary();
}
The caches are placed in the queue and are picked up by the worker thread
which just destorys them after removing from the queue.
This setup seems to generate a memory leak of about 1GB per day.
When looking at heapdump, I see all space is occupied by instances of
java.util.concurrent.ConcurrentSkipListMap$Node:
(please copy paste table into notepad to see tables correctly)
Objects by class
+------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+
| Class
| Objects | Shallow Size | Retained Size |
+------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+
| java.util.concurrent.ConcurrentSkipListMap$Node
| 4,987,415 13 % | 119,697,960 10 % | ~ 1,204,893,605 100 % |
|
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask
| 4,985,687 13 % | 239,312,976 20 % | ~ 917,361,000 76 % |
|
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner
| 4,985,680 13 % | 119,656,320 10 % | ~ 558,390,752 46 % |
| org.jsr166.ConcurrentHashMap8
| 4,990,926 13 % | 199,637,040 17 % | ~ 439,459,352 36 % |
| org.jsr166.LongAdder8
| 4,992,416 13 % | 159,757,312 13 % | ~ 159,757,312 13 % |
| org.apache.ignite.lang.IgniteUuid
| 4,989,306 13 % | 119,743,344 10 % | ~ 119,745,456 10 % |
| java.util.concurrent.ConcurrentSkipListMap$Index
| 2,488,987 7 % | 59,735,688 5 % | ~ 119,502,384 10 % |
| java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 49 0 % | 1,568 0 % | ~ 106,991,832 9 % |
| org.jsr166.ConcurrentHashMap8$ValuesView
| 4,985,368 13 % | 79,765,888 7 % | ~ 79,765,888 7 % |
| java.util.HashMap$Node
| 44,335 0 % | 1,418,720 0 % | ~ 79,618,104 7 % |
| java.util.HashMap$Node[]
| 13,093 0 % | 1,098,856 0 % | ~ 68,150,520 6 % |
| java.util.HashMap
| 13,550 0 % | 650,400 0 % | ~ 67,636,112 6 % |
| java.util.concurrent.ConcurrentSkipListMap
| 10 0 % | 480 0 % | ~ 59,830,768 5 % |
Merged paths to java.util.concurrent.ConcurrentSkipListMap$Node instances
(first 5 levels) reports no obvious dominator
(at least no dominator from my test namespace):
Merged paths
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+
| Name
| Objects | Retained Size | Dominators |
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+
| +---<All the objects>
| 4,987,415 100 % | 1,037,112,344 100 % | |
| |
| | | |
| +---<Retained from several objects simultaneously (don't have a
dominator)> | 1,245,699 25 % |
1,037,015,992 99 % | |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap
| | | |
| | | | |
| | | |
| | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap$Node
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap
| | | |
| | | |
| | | |
| | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet
| | | |
| | |
| | | |
| | +---org.apache.ignite.thread.IgniteThread
| | | |
| |
| | | |
| +---java.util.concurrent.ConcurrentSkipListMap$Node
| 3,741,257 75 % | 777,983,576 75 % | 3,741,257 |
| | |
| | | |
| | +---<Retained from several objects simultaneously (don't have a
dominator)> | 934,092 19 % |
777,912,544 75 % | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | |
| | | |
| | | +---org.apache.ignite.thread.IgniteThread
| | | |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap$Node
| 2,806,816 56 % | 583,674,344 56 % | 2,806,816 |
| | | |
| | | |
| | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 700,884 14 % |
583,622,824 56 % | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| | | |
| | | | |
| | | |
| | | | +---org.apache.ignite.thread.IgniteThread
| | | |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| 2,105,683 42 % | 437,877,768 42 % | 2,105,683 |
| | | | |
| | | |
| | | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 525,764 11 % |
437,840,192 42 % | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index
| | | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| | | |
| | | | | |
| | | |
| | | | | +---org.apache.ignite.thread.IgniteThread
| | | |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| 1,579,741 32 % | 328,509,568 32 % | 1,579,741 |
| | | | | |
| | | |
| | | | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 394,396 8 % |
328,481,960 32 % | |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node
| 1,185,216 24 % | 246,468,448 24 % | 1,185,216 |
| | | | | |
| | | |
| | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 129 0 % | 27,608 0 % | 2 |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 178 0 % | 37,576 0 % | 2 |
| | | | |
| | | |
| | | | +---java.util.concurrent.ConcurrentSkipListMap
| 178 0 % | 37,576 0 % | 2 |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 249 0 % | 51,520 0 % | 2 |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap
| 249 0 % | 51,520 0 % | 2 |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
| 249 0 % | 51,520 0 % | 2 |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 348 0 % | 71,008 0 % | 2 |
| | | |
| | | |
| | | +---java.util.concurrent.ConcurrentSkipListMap
| 348 0 % | 71,008 0 % | 2 |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
| 348 0 % | 71,008 0 % | 2 |
| | | |
| | | |
| | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 348 0 % |
71,008 0 % | |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator
| 1 0 % | 24 0 % | 1 |
| | |
| | | |
| | +---org.apache.ignite.thread.IgniteThread
| 1 0 % | 24 0 % | 1 |
| |
| | | |
| +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 458 0 % | 96,304 0 % | 11 |
| | |
| | | |
| | +---java.util.concurrent.ConcurrentSkipListMap
| 457 0 % | 96,280 0 % | 10 |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
| 451 0 % | 96,136 0 % | 2 |
| | | | |
| | | |
| | | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 451 0 % |
96,136 0 % | |
| | | | |
| | | |
| | | |
+---org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$CacheGroupHolder1
| | | |
| | | | |
| | | |
| | | |
+---org.apache.ignite.internal.processors.cache.CacheGroupContext
| | | |
| | | | |
| | | |
| | | |
+---org.apache.ignite.internal.processors.cache.GridCacheAffinityManager
| | | |
| | | |
| | | |
| | | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedSet
| 4 0 % | 96 0 % | 4 |
| | | | |
| | | |
| | | |
+---org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager
| 4 0 % | 96 0 % | 2 |
| | | | |
| | | |
| | | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 4 0 % |
96 0 % | |
| | | |
| | | |
| | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 1 0 % |
24 0 % | |
| | | | |
| | | |
| | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet
| | | |
| | | | | |
| | | |
| | | | |
+---org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor
| | | |
| | | | |
| | | |
| | | | +---org.apache.ignite.thread.IgniteThread
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
| 1 0 % | 24 0 % | 1 |
| | | |
| | | |
| | | +---<Retained from several objects simultaneously (don't have a
dominator)> | 1 0 % |
24 0 % | |
| | | |
| | | |
| | | +---java.util.LinkedList$Node
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$1
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$4
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker
| | | |
| | | |
| | | |
| | |
+---org.apache.ignite.internal.processors.cache.GridCacheSharedContext
| | | |
| | |
| | | |
| | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedMap
| 1 0 % | 24 0 % | 1 |
| | |
| | | |
| |
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
| 1 0 % | 24 0 % | 1 |
| | |
| | | |
| | +---<Retained from several objects simultaneously (don't have a
dominator)> | 1 0 % |
24 0 % | |
| | |
| | | |
| | +---java.util.LinkedList$Node
| | | |
| | |
| | | |
| |
+---org.apache.ignite.internal.processors.cache.GridCacheSharedContext
| | | |
| | |
| | | |
| |
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$2
| | | |
| | |
| | | |
| |
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$3
| | | |
| | |
| | | |
| |
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$DeadlockDetectionListener
| | | |
| |
| | | |
| +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator
| 1 0 % | 48 0 % | 1 |
| |
| | | |
| +---org.apache.ignite.thread.IgniteThread
| 1 0 % | 48 0 % | 1 |
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+
Also 2 Bigges objects - Dominators are reported as:
Objects with biggest retained size
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
|
Name
| Retained Size |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
| +---java.util.concurrent.ConcurrentSkipListMap [Stack Local] size =
4,985,687
| 59,710,656 5 % |
| |
| |
|
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
| 24,527,600 2 % |
| | |
| |
| |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeFutureSet
| 24,525,728 2 % |
| | |
| |
| | +---java.util.LinkedList size = 1,000
| 24,525,680 2 % |
I have also noticed that during the test the most active thread is always
"exchange-worker" with allocation reported by visualvm on the level of
200MB/sec at the end of the test.
Please let me know if this is a know issue - and if not - if you would be
interested in simple repro app.
Thank you in advance,
zbyszek.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
