liying created HDFS-14737:
-----------------------------
Summary: Writing data through the HDFS nfs3 service is very slow,
and timeout occurs while mount directories on the nfs client
Key: HDFS-14737
URL: https://issues.apache.org/jira/browse/HDFS-14737
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.7.2
Reporter: liying
Assignee: liying
We know the NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of
the client’s local file system. I start the portmap and nfs3server in the
hadoop node，and use the command （mount -t nfs -o vers=3,proto=tcp,nolock
nfs3serverIP:/ /hdfs） to mount. It works well。Then i ues this command to mount
on many client. And i used linux cp to copy files to the mounted
dir(/hdfs),found the speed was very slow . There is a lot of information in the
log as the follow:
2019-08-14 15:36:28,347 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,093 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,850 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,780 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,928 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:28,410 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:09,310 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:10,069 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,856 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,957 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:28,475 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,923 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,987 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:15,541 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:16,287 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:28,530 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,015 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,024 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,028 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:21,757 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:22,508 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:28,583 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,063 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,086 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:27,983 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,647 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,729 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,184 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,191 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:28,707 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
Then i print the jstack log to check the threads of nfs3server， and found some
threads' status are waiting for monitor entry : BLOCKED(on object monitor).
After checking the codes, I found that the updateMapIncr will be invoked every
time if the group or user doesn't exits. And In the class ShellBasedIdMapping,
many methods are made with synchronized. So i think we can avoid to invoke the
updateMapIncr every time.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
