TSocket timed out reading 4 bytes error against HadoopThriftServer from
Hadoop-0.20.2 in Perl
---------------------------------------------------------------------------------------------
Key: HDFS-2550
URL: https://issues.apache.org/jira/browse/HDFS-2550
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.2
Environment: Cloudera Hadoop 0.20.2 distribution running on CentOS 5.5
x64, Java SE 1.6.0_23-b05, Perl 5.14.0
Reporter: Jason Crickmer
For the past few weeks I have randomly been receiving errors via the Perl
binding of ThriftFS TSocket cannot read 4 bytes. I thought that it was a
symptom of too many clients against the server (say 16 doing a mix of file read
and write, as well as listStatus), but in the past couple of days, I have
started getting them all the time, even with only 1 client trying to read. The
Perl client error is:
{noformat}
$VAR1 = bless( {
'code' => 0,
'message' => 'TSocket: timed out reading 4 bytes from
bigwws001:9090'
}, 'Thrift::TException' );
{noformat}
This typically happens in conjunction with errors in other clients, thus
initially leading me to believe that it was really a timeout issue. But after
it started occurring with only 1 client running and well within the 10 sec
timeout I had set on the TSocket within my Perl client, I checked on the
exceptions in the ThriftFS server log:
{noformat}
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.checkBounds(String.java:397)
at java.lang.String.<init>(String.java:442)
at
org.apache.hadoop.thriftfs.HadoopThriftServer$HadoopThriftHandler.read(HadoopThriftServer.java:307)
at
org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor$read.process(Unknown
Source)
at
org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor.process(Unknown
Source)
at
com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This change seems to have "solved" the problem, but this is very much a hack
since I do not know the code (maybe throwing an IOException is the right thing
to do, and then let it turn into a ThriftIOException?).
{noformat}
---
src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java-orig
2011-11-11 09:18:44.000000000 -0600
+++
src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java
2011-11-11 09:00:47.000000000 -0600
@@ -303,8 +303,9 @@
}
byte[] tmp = new byte[length];
int numbytes = in.read(offset, tmp, 0, length);
- HadoopThriftHandler.LOG.debug("read done: " + tout.id);
- return new String(tmp, 0, numbytes, "UTF-8");
+ HadoopThriftHandler.LOG.debug("read done: " + tout.id +
+ " numbytes: " + numbytes);
+ return new String(tmp, 0, numbytes > 0 ? numbytes : 0, "UTF-8");
} catch (IOException e) {
throw new ThriftIOException(e.getMessage());
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
