[
https://issues.apache.org/jira/browse/HDDS-12497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumitra Sulav updated HDDS-12497:
----------------------------------
Description:
Steps tried :
# Created a key
# Identified the container ID
# Closed the container
# Observed the reconcile checksum output
# Deleted one of the block from the container
# Ran reconcile again
# Validate that if reconcile was able to identify the change in the block
missing/info.
The output of last reconcile should have changed and shown checksum mismatches
{code:java}
ozone sh vol create vol1
ozone sh bucket create vol1/buck1
ozone sh key put vol1/buck1/key1 LICENSE.txt -r THREE -t RATIS
ozone sh key info vol1/buck1/key1
{
"volumeName" : "vol1",
"bucketName" : "buck1",
"name" : "key1",
"owner" : "hadoop",
"dataSize" : 21937,
"creationTime" : "2025-03-07T08:06:04.902Z",
"modificationTime" : "2025-03-07T08:06:32.281Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"minimumNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"tags" : { },
"ozoneKeyLocations" : [ {
"containerID" : 2,
"localID" : 115816896921600002,
"length" : 21937,
"offset" : 0,
"keyOffset" : 0
} ],
"generation" : 11,
"file" : true
}
ozone admin container info 2
Container id: 2
Pipeline id: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
Write PipelineId: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
Write Pipeline State: OPEN
Container State: OPEN
Datanodes:
[61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default,
d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default]
Replicas: [State: OPEN; ReplicaIndex: 0; Origin:
61924ece-1789-461c-92a5-3f2739ac98d9; Location:
61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
State: OPEN; ReplicaIndex: 0; Origin: d9d707f0-7184-4b88-8514-cb256504a950;
Location:
d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default,
State: OPEN; ReplicaIndex: 0; Origin: 47b11b47-dca5-43de-9db3-6c61f6aa75cb;
Location:
47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default]
ozone admin container close 2
ozone admin container reconcile 2
Reconciliation has been triggered for container 2
Use "ozone admin container info --json 2" to see the checksums of each
container replica
bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
Deleted a block on dn3
rm
/data/hdds/hdds/CID-f4178a23-116b-42f1-9139-356f88255e06/current/containerDir0/2/chunks/115816896921600002.block
And ran the reconcile again
bash-5.1$ ozone admin container reconcile 2
Reconciliation has been triggered for container 2
Use "ozone admin container info --json 2" to see the checksums of each
container replica
bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
{code}
These tests were carried out on
https://github.com/apache/ozone/tree/HDDS-10239-container-reconciliation branch.
was:
Steps tried :
# Created a key
# Identified the container ID
# Closed the container
# Observed the reconcile checksum output
# Deleted one of the block from the container
# Ran reconcile again
# Validate that if reconcile was able to identify the change in the block
missing/info.
{code:java}
ozone sh vol create vol1
ozone sh bucket create vol1/buck1
ozone sh key put vol1/buck1/key1 LICENSE.txt -r THREE -t RATIS
ozone sh key info vol1/buck1/key1
{
"volumeName" : "vol1",
"bucketName" : "buck1",
"name" : "key1",
"owner" : "hadoop",
"dataSize" : 21937,
"creationTime" : "2025-03-07T08:06:04.902Z",
"modificationTime" : "2025-03-07T08:06:32.281Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"minimumNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"tags" : { },
"ozoneKeyLocations" : [ {
"containerID" : 2,
"localID" : 115816896921600002,
"length" : 21937,
"offset" : 0,
"keyOffset" : 0
} ],
"generation" : 11,
"file" : true
}
ozone admin container info 2
Container id: 2
Pipeline id: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
Write PipelineId: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
Write Pipeline State: OPEN
Container State: OPEN
Datanodes:
[61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default,
d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default]
Replicas: [State: OPEN; ReplicaIndex: 0; Origin:
61924ece-1789-461c-92a5-3f2739ac98d9; Location:
61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
State: OPEN; ReplicaIndex: 0; Origin: d9d707f0-7184-4b88-8514-cb256504a950;
Location:
d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default,
State: OPEN; ReplicaIndex: 0; Origin: 47b11b47-dca5-43de-9db3-6c61f6aa75cb;
Location:
47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default]
ozone admin container close 2
ozone admin container reconcile 2
Reconciliation has been triggered for container 2
Use "ozone admin container info --json 2" to see the checksums of each
container replica
bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
Deleted a block on dn3
rm
/data/hdds/hdds/CID-f4178a23-116b-42f1-9139-356f88255e06/current/containerDir0/2/chunks/115816896921600002.block
And ran the reconcile again
bash-5.1$ ozone admin container reconcile 2
Reconciliation has been triggered for container 2
Use "ozone admin container info --json 2" to see the checksums of each
container replica
bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
"dataChecksum" : "6d13c27d"
{code}
These tests were carried out on
https://github.com/apache/ozone/tree/HDDS-10239-container-reconciliation branch.
> Reconciler not working on block delete
> --------------------------------------
>
> Key: HDDS-12497
> URL: https://issues.apache.org/jira/browse/HDDS-12497
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Soumitra Sulav
> Priority: Critical
>
> Steps tried :
> # Created a key
> # Identified the container ID
> # Closed the container
> # Observed the reconcile checksum output
> # Deleted one of the block from the container
> # Ran reconcile again
> # Validate that if reconcile was able to identify the change in the block
> missing/info.
> The output of last reconcile should have changed and shown checksum mismatches
> {code:java}
> ozone sh vol create vol1
> ozone sh bucket create vol1/buck1
> ozone sh key put vol1/buck1/key1 LICENSE.txt -r THREE -t RATIS
> ozone sh key info vol1/buck1/key1
> {
> "volumeName" : "vol1",
> "bucketName" : "buck1",
> "name" : "key1",
> "owner" : "hadoop",
> "dataSize" : 21937,
> "creationTime" : "2025-03-07T08:06:04.902Z",
> "modificationTime" : "2025-03-07T08:06:32.281Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "ozoneKeyLocations" : [ {
> "containerID" : 2,
> "localID" : 115816896921600002,
> "length" : 21937,
> "offset" : 0,
> "keyOffset" : 0
> } ],
> "generation" : 11,
> "file" : true
> }
> ozone admin container info 2
> Container id: 2
> Pipeline id: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
> Write PipelineId: 2143b678-e089-4eb0-af0a-fe9c3fcdce91
> Write Pipeline State: OPEN
> Container State: OPEN
> Datanodes:
> [61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
> 47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default,
> d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default]
> Replicas: [State: OPEN; ReplicaIndex: 0; Origin:
> 61924ece-1789-461c-92a5-3f2739ac98d9; Location:
> 61924ece-1789-461c-92a5-3f2739ac98d9/ozone-ha-datanode-2.ozone-ha_default,
> State: OPEN; ReplicaIndex: 0; Origin: d9d707f0-7184-4b88-8514-cb256504a950;
> Location:
> d9d707f0-7184-4b88-8514-cb256504a950/ozone-ha-datanode-3.ozone-ha_default,
> State: OPEN; ReplicaIndex: 0; Origin: 47b11b47-dca5-43de-9db3-6c61f6aa75cb;
> Location:
> 47b11b47-dca5-43de-9db3-6c61f6aa75cb/ozone-ha-datanode-1.ozone-ha_default]
> ozone admin container close 2
> ozone admin container reconcile 2
> Reconciliation has been triggered for container 2
> Use "ozone admin container info --json 2" to see the checksums of each
> container replica
> bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
> "dataChecksum" : "6d13c27d"
> "dataChecksum" : "6d13c27d"
> "dataChecksum" : "6d13c27d"
> Deleted a block on dn3
> rm
> /data/hdds/hdds/CID-f4178a23-116b-42f1-9139-356f88255e06/current/containerDir0/2/chunks/115816896921600002.block
> And ran the reconcile again
> bash-5.1$ ozone admin container reconcile 2
> Reconciliation has been triggered for container 2
> Use "ozone admin container info --json 2" to see the checksums of each
> container replica
> bash-5.1$ ozone admin container info --json 2 | grep -i "checksum"
> "dataChecksum" : "6d13c27d"
> "dataChecksum" : "6d13c27d"
> "dataChecksum" : "6d13c27d"
> {code}
> These tests were carried out on
> https://github.com/apache/ozone/tree/HDDS-10239-container-reconciliation
> branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
