Hi, I am trying to run hadoop with ceph as the backend. I installed the libcephfs-jni and libcephfs-java to get the libcephfs.jar and the related .so libraries. Also I compiled the cephfs-hadoop-1.0-SNAPSHOT.jar from
https://github.com/GregBowyer/cephfs-hadoop since this was the only jar which had CephHadoop2FileSystem which was need to make ceph work with yarn. I am able to browse the ceph through the hdfs commands e.g 'hdfs dfs -ls /' and also read files from ceph such as 'hdfs dfs -cat /xxxx', even the mkdir works, but when I try to write a file, it fails with the following exception 14/10/01 16:26:17 FATAL distributedshell.Client: Error running CLient java.io.IOException: Operation not permitted at com.ceph.fs.CephMount.native_ceph_open(Native Method) at com.ceph.fs.CephMount.open(CephMount.java:545) at org.apache.hadoop.fs.ceph.CephTalker.__open(CephTalker.java:158) at org.apache.hadoop.fs.ceph.CephFileSystem.selectDataPool(CephFileSystem.java:308) at org.apache.hadoop.fs.ceph.CephFileSystem.create(CephFileSystem.java:433) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1903) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1871) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1836) at org.apache.hadoop.yarn.applications.distributedshell.Client.addToLocalResources(Client.java:756) at org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:471) at org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:198) >From a casual browse of the code, it seems that the error is generated from public class CephFileSystem extends FileSystem { private String selectDataPool(Path path, int repl_wanted) throws IOException { int fd = ceph.__open(new Path("/"), CephMount.O_RDONLY, 0); My core-site.xml looks like <property> <name>fs.defaultFS</name> <value>ceph://XX.XX.XX.XX:6789/</value> </property> <property> <name>hadoop.sharedtmp.dir1</name> <value>/mnt/mycephfs/hadoop-tmp/hadoop-${user.name}</value> </property> <property> <name>fs.ceph.impl</name> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.ceph.impl</name> <value>org.apache.hadoop.fs.ceph.CephHadoop2FileSystem</value> </property> <property> <name>ceph.conf.file</name> <value>/etc/ceph/ceph.conf</value> </property> <property> <name>ceph.auth.id</name> <value>admin</value> </property> <property> <name>ceph.auth.keyring</name> <value>/etc/ceph/ceph.client.admin.keyring</value> </property> <property> <name>ceph.data.pools</name> <value>hadoop</value> </property> Thanks in advance for any pointers. Gurmeet
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
