Jinglun created HDFS-14633:
------------------------------
Summary: The StorageType consume in QuotaFeature is not handled
when addBlock, delete, rename, setStoragePolicy etc.
Key: HDFS-14633
URL: https://issues.apache.org/jira/browse/HDFS-14633
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Jinglun
Assignee: Jinglun
The NameNode manages the global state of the cluster. We should always take
NameNode's records as the sole criterion because no matter what inconsistent is
the NameNode should finally make everything right based on it's records. Let's
call it rule NSC(NameNode is the Sole Criterion). That means when we do all
quota related rpcs, we do the quota check according to NameNode's records
regardless of any inconsistent situation, such as the replicas doesn't match
the storage policy of the file, or the replicas count doesn't match the file's
set replica.
The work SPS deals with the wrongly palced replicas. Someone might think about
putting off the consume update of the DirectoryQuota until all replicas are
re-placed by SPS. I can't agree that because if we do so we will abandon
letting the NameNode's records to be the sole criterion. The block replication
is a good example of the rule NSC. When we count the consume of a
file(CONTIGUOUS), we multiply the replication factor with the file's length, no
matter the blocks are under replicated or excessed. We should do the same thing
for the storage type quota.
Another concern is the change will let setStoragePolicy throw
QuotaByStorageTypeExceededException which it doesn't before. I don't think it's
a big problem since the setStoragePolicy already throws IOException. Or we can
wrap the QuotaByStorageTypeExceededException in an IOException, but I won't
recommend that because it's ugly.
To make storage type consume follow the rule NSC, we must change every rpc
related to consume changing, such as addBlock, delete, rename(especially moving
a file with storage policy inherited from it's parent)， setStoragePolicy,
setReplication etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
