takeshi.miao created HDFS-4881:
----------------------------------
Summary: fine tune "Access token verification failed" error msg in
datanode log
Key: HDFS-4881
URL: https://issues.apache.org/jira/browse/HDFS-4881
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 1.0.0
Environment: CentOS-5.3, java-version-1.6.0_26
Reporter: takeshi.miao
Priority: Trivial
I'd like to issue this ticket is due to we suffered a datanode access token
verification failure issue recently. The client is HBase who is accessing the
local datanode via DFSClient. The details log snippets as follows...
*regionserver log*
{code}
...
[2013-05-24 08:33:37,553][regionserver8120-compactions-1369288874174][INFO
][org.apache.hadoop.hbase.regionserver.Store]: Started compaction of 1 file(s)
in cf=ho, hasReferences=true, into
hdfs://sjdc-s-hdd-001.sjdc.ispn.trendmicro.com:8020/user/SPN-hbase/spn.guidcensus.ho/f99c6fb26f488034bf0e6ddd7a647ba4/.tmp,
seqid=3, totalSize=4.2g
[2013-05-24 08:33:37,554][regionserver8120-compactions-1369288874174][INFO
][org.apache.hadoop.hdfs.DFSClient]: Access token was invalid when connecting
to /10.31.6.49:1004 :
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
access token error for OP_READ_BLOCK, self=/10.31.6.49:36530,
remote=/10.31.6.49:1004, for file
/user/SPN-hbase/spn.guidcensus.ho/a565dd142933e3abf9bec33d59210d1b/ho/c5b37b9dd8801275c8fb160c0fb32ce5c48b56f4,
for block 4549293737579979499_205814042
...
{code}
*datanode log*
{code}
...
[2013-05-24 08:33:37,554][DataXceiver for client /10.31.6.49:36530 [Waiting for
operation #1]][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode]:
DatanodeRegistration(10.31.6.49:1004, storageID=DS-1953102179-10.31.6.49-1004-
1342490559943, infoPort=1006, ipcPort=50020):DataXceiver
java.io.IOException: Access token verification failed, for client
/10.31.6.49:36530 for OP_READ_BLOCK for block blk_4549293737579979499_205814042
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:252)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:175)
...
{code}
After trace o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java, I
found that there are more further details error description written in code.
*o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java*
{code}
public void checkAccess(BlockTokenIdentifier id, String userId, Block block,
AccessMode mode) throws InvalidToken {
if (LOG.isDebugEnabled()) {
LOG.debug("Checking access for user=" + userId + ", block=" + block
+ ", access mode=" + mode + " using " + id.toString());
}
if (userId != null && !userId.equals(id.getUserId())) {
throw new InvalidToken("Block token with " + id.toString()
+ " doesn't belong to user " + userId);
}
if (id.getBlockId() != block.getBlockId()) {
throw new InvalidToken("Block token with " + id.toString()
+ " doesn't apply to block " + block);
}
if (isExpired(id.getExpiryDate())) {
throw new InvalidToken("Block token with " + id.toString()
+ " is expired.");
}
if (!id.getAccessModes().contains(mode)) {
throw new InvalidToken("Block token with " + id.toString()
+ " doesn't have " + mode + " permission");
}
}
{code}
But actually, this InvalidTokenException will not be handled further (but
caught), so I can not trace what kind of this access block token verification
is...
*o.a.h.hdfs.server.datanode.DataXceiver.java*
{code}
...
if (datanode.isBlockTokenEnabled) {
try {
datanode.blockTokenSecretManager.checkAccess(accessToken, null, block,
BlockTokenSecretManager.AccessMode.READ);
} catch (InvalidToken e) {
// the e object not handled further...
try {
out.writeShort(DataTransferProtocol.OP_STATUS_ERROR_ACCESS_TOKEN);
out.flush();
throw new IOException("Access token verification failed, for client "
+ remoteAddress + " for OP_READ_BLOCK for block " + block);
} finally {
IOUtils.closeStream(out);
}
}
}
...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
