Hi David,
Thanks for responding! My main intention was to submit spark Job/jar to
yarn cluster from my eclipse with in the code. Is there any way that I
could pass my yarn configuration somewhere in the code to submit the jar to
the cluster?
Thank you,
Ajay
On Sunday, August 30, 2015, David Mitchell <jdavidmitch...@gmail.com> wrote:
> Hi Ajay,
>
> Are you trying to save to your local file system or to HDFS?
>
> // This would save to HDFS under "/user/hadoop/counter"
> counter.saveAsTextFile("/user/hadoop/counter");
>
> David
>
>
> On Sun, Aug 30, 2015 at 11:21 AM, Ajay Chander <itsche...@gmail.com
> <javascript:_e(%7B%7D,'cvml','itsche...@gmail.com');>> wrote:
>
>> Hi Everyone,
>>
>> Recently we have installed spark on yarn in hortonworks cluster. Now I am
>> trying to run a wordcount program in my eclipse and I
>> did setMaster("local") and I see the results that's as expected. Now I want
>> to submit the same job to my yarn cluster from my eclipse. In storm
>> basically I was doing the same by using StormSubmitter class and by passing
>> nimbus & zookeeper host to Config object. I was looking for something
>> exactly the same.
>>
>> When I went through the documentation online, it read that I am suppose
>> to "export HADOOP_HOME_DIR=path to the conf dir". So now I copied the conf
>> folder from one of sparks gateway node to my local Unix box. Now I did
>> export that dir...
>>
>> export HADOOP_HOME_DIR=/Users/user1/Documents/conf/
>>
>> And I did the same in .bash_profile too. Now when I do echo
>> $HADOOP_HOME_DIR, I see the path getting printed in the command prompt. Now
>> my assumption is, in my program when I change setMaster("local") to
>> setMaster("yarn-client") my program should pick up the resource mangers i.e
>> yarn cluster info from the directory which I have exported and the job
>> should get submitted to resolve manager from my eclipse. But somehow it's
>> not happening. Please tell me if my assumption is wrong or if I am missing
>> anything here.
>>
>> I have attached the word count program that I was using. Any help is
>> highly appreciated.
>>
>> Thank you,
>> Ajay
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> <javascript:_e(%7B%7D,'cvml','user-unsubscr...@spark.apache.org');>
>> For additional commands, e-mail: user-h...@spark.apache.org
>> <javascript:_e(%7B%7D,'cvml','user-h...@spark.apache.org');>
>>
>
>
>
> --
> ### Confidential e-mail, for recipient's (or recipients') eyes only, not
> for distribution. ###
>
