create a directory "/home/username/hdfs" (or at some place of your
choice)..inside this hdfs directory create three sub directories -
name, data, and temp, then follow these steps :
add following properties in your core-site.xml -
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/mohammad/hdfs/temp</value>
</property>
then add following two properties in your hdfs-site.xml -
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/mohammad/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/mohammad/hdfs/data</value>
</property>
finally add this property in your mapred-site.xml -
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9001</value>
</property>
NOTE: you can give any name to these directories of your choice, just
keep in mind you have to give same names as values of
above specified properties in your configuration files.
(give full path of these directories, not just the name of the
directory)
After this follow the steps provided in the previous reply.
Regards,
Mohammad Tariq
On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan <babak...@gmail.com> wrote:
> thank's Mohammad
>
> with this command:
>
> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>
> this is my output:
>
> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = ubuntu/127.0.1.1
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
> 12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0
> seconds.
> 12/06/06 20:05:20 INFO common.Storage: Storage directory
> /tmp/hadoop-babak/dfs/name has been successfully formatted.
> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
> ************************************************************/
>
> by this command:
>
> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
>
> this is the out put
>
> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
> anlegen: Keine Berechtigung
>
> this out put(it's in german and it means no right to make this folder)
>
>
> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq <donta...@gmail.com> wrote:
>>
>> once we are done with the configuration, we need to format the file
>> system..use this command to do that-
>> bin/hadoop namenode -format
>>
>> after this, hadoop daemon processes should be started using following
>> commands -
>> bin/start-dfs.sh (it'll start NN & DN)
>> bin/start-mapred.sh (it'll start JT & TT)
>>
>> after this use jps to check if everything is alright or point your
>> browser to localhost:50070..if you further find any problem provide us
>> with the error logs..:)
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan <babak...@gmail.com> wrote:
>> > were you able to format hdfs properly???
>> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
>> > install
>> > Hadoop?
>> >
>> > On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq <donta...@gmail.com>
>> > wrote:
>> >>
>> >> if you are getting only this, it means your hadoop is not
>> >> running..were you able to format hdfs properly???
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan <babak...@gmail.com>
>> >> wrote:
>> >> > Hi MohammadmI irun jps in my shel I can see this result:
>> >> > 2213 Jps
>> >> >
>> >> >
>> >> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq <donta...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> you can also use "jps" command at your shell to see whether Hadoop
>> >> >> processes are running or not.
>> >> >>
>> >> >> Regards,
>> >> >> Mohammad Tariq
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq <donta...@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Babak,
>> >> >> >
>> >> >> > You have to type it in you web browser..Hadoop provides us a web
>> >> >> > GUI
>> >> >> > that not only allows us to browse through the file system, but to
>> >> >> > download the files as well..Apart from that it also provides a web
>> >> >> > GUI
>> >> >> > that can be used to see the status of Jobtracker and
>> >> >> > Tasktracker..When
>> >> >> > you run a Hive or Pig job or a Mapreduce job, you can point your
>> >> >> > browser to http://localhost:50030 to see the status and logs of
>> >> >> > your
>> >> >> > job.
>> >> >> >
>> >> >> > Regards,
>> >> >> > Mohammad Tariq
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan <babak...@gmail.com>
>> >> >> > wrote:
>> >> >> >> Thank you shashwat for the answer,
>> >> >> >> where should I type http://localhost:50070?
>> >> >> >> I typed here: hive>http://localhost:50070 but nothing as result
>> >> >> >>
>> >> >> >>
>> >> >> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
>> >> >> >> <dwivedishash...@gmail.com> wrote:
>> >> >> >>>
>> >> >> >>> first type http://localhost:50070 whether this is opening or not
>> >> >> >>> and
>> >> >> >>> check
>> >> >> >>> how many nodes are available, check some of the hadoop shell
>> >> >> >>> commands
>> >> >> >>>
>> >> >> >>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html
>> >> >> >>> run
>> >> >> >>> example mapreduce task on hadoop take example from here
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> : http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>> >> >> >>>
>> >> >> >>> if all the above you can do sucessfully means hadoop is
>> >> >> >>> configured
>> >> >> >>> correctly
>> >> >> >>>
>> >> >> >>> Regards
>> >> >> >>> Shashwat
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan
>> >> >> >>> <babak...@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>>
>> >> >> >>>> no I'm not working on CDH.Is there a way to test if my Hadoop
>> >> >> >>>> works
>> >> >> >>>> fine
>> >> >> >>>> or not?
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS <bejoy...@yahoo.com>
>> >> >> >>>> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Hi Babak
>> >> >> >>>>>
>> >> >> >>>>> You gotta follow those instructions in the apace site to set
>> >> >> >>>>> up
>> >> >> >>>>> hadoop
>> >> >> >>>>> from scratch and ensure that hdfs is working first. You should
>> >> >> >>>>> be
>> >> >> >>>>> able to
>> >> >> >>>>> read and write files to hdfs before you do your next steps.
>> >> >> >>>>>
>> >> >> >>>>> Are you on CDH or apache distribution of hadoop? If it is CDH
>> >> >> >>>>> there
>> >> >> >>>>> are
>> >> >> >>>>> detailed instructions on Cloudera web site.
>> >> >> >>>>>
>> >> >> >>>>> Regards
>> >> >> >>>>> Bejoy KS
>> >> >> >>>>>
>> >> >> >>>>> Sent from handheld, please excuse typos.
>> >> >> >>>>> ________________________________
>> >> >> >>>>> From: Babak Bastan <babak...@gmail.com>
>> >> >> >>>>> Date: Tue, 5 Jun 2012 21:30:22 +0200
>> >> >> >>>>> To: <user@hive.apache.org>
>> >> >> >>>>> ReplyTo: user@hive.apache.org
>> >> >> >>>>> Subject: Re: Error while Creating Table in Hive
>> >> >> >>>>>
>> >> >> >>>>> @Bejoy: I set the fs.default.name in the core-site.xml and I
>> >> >> >>>>> did
>> >> >> >>>>> all
>> >> >> >>>>> of
>> >> >> >>>>> thing that was mentioned in the reference but no effect
>> >> >> >>>>>
>> >> >> >>>>> On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan
>> >> >> >>>>> <babak...@gmail.com>
>> >> >> >>>>> wrote:
>> >> >> >>>>>>
>> >> >> >>>>>> Ok sorry but that was my Mistake .I thought it works but no.
>> >> >> >>>>>> I wrote the command without ; and then I think It works but
>> >> >> >>>>>> with
>> >> >> >>>>>> ;
>> >> >> >>>>>> at
>> >> >> >>>>>> the end of command
>> >> >> >>>>>>
>> >> >> >>>>>> CREATE TABLE pokes (foo INT, bar STRING);
>> >> >> >>>>>>
>> >> >> >>>>>> does'nt work
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
>> >> >> >>>>>> <dwivedishash...@gmail.com> wrote:
>> >> >> >>>>>>>
>> >> >> >>>>>>> inside configuration. all properties will be inside the
>> >> >> >>>>>>> configuration
>> >> >> >>>>>>> tags
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan
>> >> >> >>>>>>> <babak...@gmail.com>
>> >> >> >>>>>>> wrote:
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Thank you so much my friend your idee works fine(no error)
>> >> >> >>>>>>>> you
>> >> >> >>>>>>>> are
>> >> >> >>>>>>>> the best :)
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan
>> >> >> >>>>>>>> <babak...@gmail.com>
>> >> >> >>>>>>>> wrote:
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> It must be inside the <configuration></configuration> or
>> >> >> >>>>>>>>> outside
>> >> >> >>>>>>>>> this?
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
>> >> >> >>>>>>>>> <dwivedishash...@gmail.com> wrote:
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> It will be inside hive/conf
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan
>> >> >> >>>>>>>>>> <babak...@gmail.com>
>> >> >> >>>>>>>>>> wrote:
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> Thanks sShashwat, and where is this hive-site.xml
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv
>> >> >> >>>>>>>>>>> <dwivedishash...@gmail.com> wrote:
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> set
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> hive.metastore.warehouse.dir in hive-site.xml
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> <property>
>> >> >> >>>>>>>>>>>> <name>hive.metastore.local</name>
>> >> >> >>>>>>>>>>>> <value>true</value>
>> >> >> >>>>>>>>>>>> </property>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> <name>hive.metastore.warehouse.dir</name>
>> >> >> >>>>>>>>>>>> <value>/home/<your
>> >> >> >>>>>>>>>>>> username>/hivefolder</value>
>> >> >> >>>>>>>>>>>> <description>location of default
>> >> >> >>>>>>>>>>>> database
>> >> >> >>>>>>>>>>>> for
>> >> >> >>>>>>>>>>>> the
>> >> >> >>>>>>>>>>>> warehouse</description>
>> >> >> >>>>>>>>>>>> </property>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan
>> >> >> >>>>>>>>>>>> <babak...@gmail.com> wrote:
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> Hello Experts ,
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> I'm new in Hive .When try to create a test Table in
>> >> >> >>>>>>>>>>>>> Hive
>> >> >> >>>>>>>>>>>>> I
>> >> >> >>>>>>>>>>>>> get
>> >> >> >>>>>>>>>>>>> an error.I want to run this command:
>> >> >> >>>>>>>>>>>>> CREATE TABLE Test (DateT STRING, Url STRING, Content
>> >> >> >>>>>>>>>>>>> STRING);
>> >> >> >>>>>>>>>>>>> but this error occured:
>> >> >> >>>>>>>>>>>>> FAILED: Error in metadata: MetaException(message:Got
>> >> >> >>>>>>>>>>>>> exception:
>> >> >> >>>>>>>>>>>>> java.io.FileNotFoundException File
>> >> >> >>>>>>>>>>>>> file:/user/hive/warehouse/test does not
>> >> >> >>>>>>>>>>>>> exist.)
>> >> >> >>>>>>>>>>>>> FAILED: Execution Error, return code 1 from
>> >> >> >>>>>>>>>>>>> org.apache.hadoop.hive.ql.exec.DDLTask
>> >> >> >>>>>>>>>>>>> How can I solve this Problem?
>> >> >> >>>>>>>>>>>>> Thank you so much
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> --
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> ∞
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> Shashwat Shriparv
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> --
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> ∞
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> Shashwat Shriparv
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>> --
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>> ∞
>> >> >> >>>>>>>
>> >> >> >>>>>>> Shashwat Shriparv
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> ∞
>> >> >> >>>
>> >> >> >>> Shashwat Shriparv
>> >> >> >>>
>> >> >> >>>
>> >> >> >>
>> >> >
>> >> >
>> >
>> >
>
>
