Maybe the warnings are not the cause but a consequence (gc calls finalize() and not the other way around)
Any details on memory usage? G1? Non full Vs full gc ratio, average freed memory... Did any gc run in the 3rd RS? Memory percentage assigned to memstores? You have an average memory assigned of 120MB per region assuming 1 column family per table, and HBase default memstore size is already 256mb. You might just being running oom. Read/write pattern? Is there an intensive write? In my experience these kind of issues are caused by a bad HBase memory tuning. On 1 Feb 2018 02:41, "Jins George" <jins.geo...@aeris.net> wrote: Hi, On analyzing a prod issue of High CPU usage on Hbase Region server, I came across warning messages from region server logs complaining about Orphaned chunk of memory. 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of 104000 bytes found during finalize 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of 104000 bytes found during finalize The high CPU usage looks like due to garbage collection and it lasted for almost 6 hours. And throughout 6 hours, region server logs had these warning messages logged. Cluster Details: 4 node( 1 master + 3 slaves) cdh cluster Hbase version 1.2 Phoenix version 4.7 Region Server Heap : 4G Total Regions: ~135 Total tables : ~35 Out of 3 region servers, 2 of them had the warning logs and both suffered high CPU. Third region server nither had High CPU nor the warning logs. Any idea why these messages are logged and can that trigger continuous GC ? Before this issue started( or around the same time) huge application log files were copied to HDFS by Yarn.. But can't think of that causing issue on Hbase Region server. Any help is appreciated. Thanks, Jins George
