Hi,
On analyzing a prod issue of High CPU usage on Hbase Region server, I
came across warning messages from region server logs complaining about
Orphaned chunk of memory.
2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager:
Orphaned chunk of 104000 bytes found during finalize
2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager:
Orphaned chunk of 104000 bytes found during finalize
The high CPU usage looks like due to garbage collection and it lasted
for almost 6 hours. And throughout 6 hours, region server logs had
these warning messages logged.
Cluster Details:
4 node( 1 master + 3 slaves) cdh cluster
Hbase version 1.2
Phoenix version 4.7
Region Server Heap : 4G
Total Regions: ~135
Total tables : ~35
Out of 3 region servers, 2 of them had the warning logs and both
suffered high CPU. Third region server nither had High CPU nor the
warning logs. Any idea why these messages are logged and can that
trigger continuous GC ?
Before this issue started( or around the same time) huge application log
files were copied to HDFS by Yarn.. But can't think of that causing
issue on Hbase Region server.
Any help is appreciated.
Thanks,
Jins George
