Re: [ceph-users] guidance with hadoop on ceph

2013-03-21 Thread Noah Watkins
:) > > > François. > > > > From: jayh...@cs.ucsc.edu > > Date: Tue, 19 Mar 2013 11:08:12 -0700 > > To: varun@gmail.com > > CC: ceph-users@lists.ceph.com > > Subject: Re: [ceph-users] guidance with hadoop on ceph > > > > No problem! Let us know if

Re: [ceph-users] guidance with hadoop on ceph

2013-03-21 Thread François P-L
ation :) François. > From: jayh...@cs.ucsc.edu > Date: Tue, 19 Mar 2013 11:08:12 -0700 > To: varun@gmail.com > CC: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] guidance with hadoop on ceph > > No problem! Let us know if you have any other issues. > > -Noah &

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
No problem! Let us know if you have any other issues. -Noah On Mar 19, 2013, at 11:05 AM, Varun Chandramouli wrote: > Hi Noah, > > Setting LD_LIBRARY_PATH in conf/hadoop-env.sh seems to have done the trick. > Thanks a ton. > > Varun > ___ ceph-us

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
Hi Noah, Setting LD_LIBRARY_PATH in conf/hadoop-env.sh seems to have done the trick. Thanks a ton. Varun ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
Are you setting LD_LIBRARY_PATH in your bashrc? If so, make sure it is set at the _very_ top (before the handling for interactive mode, a common problem with stock Ubuntu setups). Alternatively, set LD_LIBRARY_PATH in conf/hadoop-env.sh. -Noah On Mar 19, 2013, at 10:32 AM, Varun Chandramouli

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
Task tracker is running on both the nodes, and is giving the same error on both of them (and I am able to run hadoop fs -ls on both). I am running hadoop as the same user, varunc, and I guess LD_LIBRARY_PATH has been setup correctly: varunc@varunc4-virtual-machine:/usr/local/hadoop$ echo $LD_LIBRA

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
On Mar 19, 2013, at 10:05 AM, Varun Chandramouli wrote: > libcephfs_jni.so is present in /usr/local/lib/, which I added to > LD_LIBRARY_PATH and tried it again. The same error is displayed in the log > file for the task trackers. Anything else I should be doing? It looks like something is con

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
Hi, The job tracker seems to be running fine. The task tracker however is shutting down with the error: 2013-03-19 22:29:24,583 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.lang.UnsatisfiedLinkError: no cephfs_jni in java.library.path libcephfs_jni.so is pre

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
Getting closer! I suggest checking the log files for your job tracker and all of your task tracker nodes to see if any of them are having troubles. -Noah On Mar 19, 2013, at 8:42 AM, Varun Chandramouli wrote: > No, hadoop data is not located in /mnt/ceph/wc, but now i copied the data > into /

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
No, hadoop data is not located in /mnt/ceph/wc, but now i copied the data into /user/varunc/wc, which cntains the hadoop data. However, the MR-wordcount example still is not working. Its stuck with this output: varunc@varunc4-virtual-machine:~$ time /usr/local/hadoop/bin/hadoop jar hadoop*examples

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
On Mar 19, 2013, at 8:10 AM, Varun Chandramouli wrote: > Hi Noah, > > After removing that and running, I get this error: > > varunc@varunc4-virtual-machine:~$ time /usr/local/hadoop/bin/hadoop jar > hadoop*examples*.jar wordcount /mnt/ceph/wc /mnt/ceph/op2 > Warning: $HADOOP_HOME is deprecate

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
Varun, Can you post your updated core-site.xml? I've seen this error before when I used the URI: ceph:// as opposed to ceph:///. - Noah On Mar 19, 2013, at 8:10 AM, Varun Chandramouli wrote: > Hi Noah, > > After removing that and running, I get this error: > > varunc@varunc4-virtual-machin

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
Hi Noah, After removing that and running, I get this error: varunc@varunc4-virtual-machine:~$ time /usr/local/hadoop/bin/hadoop jar hadoop*examples*.jar wordcount /mnt/ceph/wc /mnt/ceph/op2 Warning: $HADOOP_HOME is deprecated. 13/03/19 20:35:03 INFO ceph.CephFileSystem: selectDataPool path=ceph:

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Noah Watkins
Hi Varun, Try removing this configuration option: > > ceph.root.dir > /mnt/ceph > Hadoop running on Ceph uses the libcephfs user-space library to talk directly to the file system, as opposed to running through the kernel or FUSE client. This setting is which directory within the Ceph fil

Re: [ceph-users] guidance with hadoop on ceph

2013-03-19 Thread Varun Chandramouli
Hi, Sorry for bringing this thread up again. After building hadoop and ceph, I am not able to run the wordcount example. I am getting the following error: varunc@varunc4-virtual-machine:/usr/local/hadoop$ time /usr/local/hadoop/bin/hadoop jar hadoop*examples*.jar wordcount /mnt/ceph/wc2 /mnt/ceph