Yiqun Lin created HDFS-13692:
--------------------------------
Summary: StorageInfoDefragmenter floods log when compacting
StorageInfoTreeset
Key: HDFS-13692
URL: https://issues.apache.org/jira/browse/HDFS-13692
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.1.0
Reporter: Yiqun Lin
StorageInfoDefragmenter floods log when compacting StorageInfoTreeset. In
{{StorageInfoDefragmenter#scanAndCompactStorages}}, it prints for all the
StorageInfo under each DN. If there are 1k nodes in cluster, and each node has
10 data dir configured, it will print 10k lines every compact interval time (10
mins). The log looks large, we could switch log level from INFO to DEBUG in
{{StorageInfoDefragmenter#scanAndCompactStorages}}.
{noformat}
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 : 0.8222276264591439
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 : 0.9330040998881849
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb : 0.9314626719970249
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd : 0.9309580852251582
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 : 0.8938870614035088
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa : 0.8963103205353998
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 : 0.8950508004926109
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 : 0.8947356866877415
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
