Manoj Govindassamy created HDFS-11382:
-----------------------------------------
Summary: Persist Erasure Coding Policy ID in a new optional field
in INodeFile in FSImage
Key: HDFS-11382
URL: https://issues.apache.org/jira/browse/HDFS-11382
Project: Hadoop HDFS
Issue Type: Task
Components: hdfs
Affects Versions: 3.0.0-alpha1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
For Erasure Coded files, replication field in INodeFile message is re-used to
store the EC Policy ID.
*FSDirWriteFileOp#addFile*
{noformat}
private static INodesInPath addFile(
FSDirectory fsd, INodesInPath existing, byte[] localName,
PermissionStatus permissions, short replication, long preferredBlockSize,
String clientName, String clientMachine)
throws IOException {
.. .. ..
try {
ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
if (ecPolicy != null) {
replication = ecPolicy.getId(); <===
}
final BlockType blockType = ecPolicy != null?
BlockType.STRIPED : BlockType.CONTIGUOUS;
INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
modTime, modTime, replication, preferredBlockSize, blockType);
newNode.setLocalName(localName);
newNode.toUnderConstruction(clientName, clientMachine);
newiip = fsd.addINode(existing, newNode, permissions.getPermission());
{noformat}
With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly
getting the EC ID from the replication field and then uses the right Policy to
construct the blocks.
*FSImageFormatPBINode#Loader#loadInodeFile*
{noformat}
ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
null;
{noformat}
The original intention was to re-use the replication field so the in-memory
representation would be compact. But, this isn't necessary for the on-disk
representation. replication is an optional field, and if we add another
optional field for the EC policy, it won't be any extra space.
Also, we need to make sure to have the appropriate asserts in place to make
sure both fields aren't set for the same INodeField.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
