Yongjun Zhang created HDFS-10667:
------------------------------------
Summary: Report more accurate info about data corruption location
Key: HDFS-10667
URL: https://issues.apache.org/jira/browse/HDFS-10667
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs
Reporter: Yongjun Zhang
Per
https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15376897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15376897
129.77 report:
{code}
2016-07-13 11:49:01,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving blk_1116167880_42906656 src: /10.6.134.229:43844 dest:
/10.6.129.77:5080
2016-07-13 11:49:01,543 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Checksum error in block blk_1116167880_42906656 from /10.6.134.229:43844
org.apache.hadoop.fs.ChecksumException: Checksum error:
DFSClient_NONMAPREDUCE_2019484565_1 at 81920 exp: 1352119728 got: -1012279895
at
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native
Method)
at
org.apache.hadoop.util.NativeCrc32.verifyChunkedSumsByteArray(NativeCrc32.java:69)
at
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:347)
at
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:294)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:421)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:558)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:789)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:917)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:174)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:80)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
at java.lang.Thread.run(Thread.java:745)
2016-07-13 11:49:01,543 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for blk_1116167880_42906656
java.io.IOException: Terminating due to a checksum error.java.io.IOException:
Unexpected checksum mismatch while writing blk_1116167880_42906656 from
/10.6.134.229:43844
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:571)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:789)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:917)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:174)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:80)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
at java.lang.Thread.run(Thread.java:745)
{code}
and
https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15378879&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15378879
{quote}
While verifying only packet, the position mentioned in the checksum exception,
is relative to packet buffer offset, not the block offset. So 81920 is the
offset in the exception.
{quote}
Create this jira to report more accurate corruption location information: the
offset in the file, offset in block, and offset in packet.
See
https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15387083&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15387083
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
