Ok, so back to basics. correct me If I am wrong
You are trying to setup YARN using s3 as file system? Then you want to use hive on top of it. On Tue, Sep 10, 2013 at 4:48 PM, <shouvanik.hal...@accenture.com> wrote: > The issue is if I paste it in core-site.xml of Hadoop on the node in > question, then will starting datanode would do only? Actually,. I am a bit > confused what all daemons to start. I have a YARN architecture running 1 > separate – Resource Manager, 1 NameNode & 4 datanodes.**** > > ** ** > > *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com] > *Sent:* Tuesday, September 10, 2013 4:43 PM > > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > ** ** > > There is not more to start. If hive is not starting properly, it will not > start at all so you are doing ok. **** > > ** ** > > Problem is when you create table, hive tries to create a directory with > that name if its not existing. Just try to put those two settings inside > core-site.xml and see if the hdfs can use it to create directory **** > > ** ** > > On Tue, Sep 10, 2013 at 4:06 PM, <shouvanik.hal...@accenture.com> wrote:** > ** > > The values are showing when I run “hive> set;” command. But it’s not > getting identified. I guess I am not starting HIVE properly. Do you know > what are the steps?**** > > **** > > *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com] > *Sent:* Tuesday, September 10, 2013 4:04 PM**** > > > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > From the config file it looks ok. Not really sure why it is not picking up > when you run it via hive cli. **** > > **** > > Can you try putting these in core-site.xml of hdfs. **** > > **** > > **** > > <property>**** > > <name>fs.s3n.awsAccessKeyId</name>**** > > <value>ID</value>**** > > </property>**** > > <property>**** > > <name>fs.s3n.awsSecretAccessKey</name>**** > > <value>SECRET</value>**** > > </property>**** > > **** > > On Tue, Sep 10, 2013 at 3:11 PM, <shouvanik.hal...@accenture.com> wrote:** > ** > > Hi Nitin,**** > > **** > > PFA hive-site.xml. I wanted to start Hortonworks HIVE. Can you please help. > **** > > **** > > Thanks,**** > > Shouvanik**** > > **** > > *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com] > *Sent:* Tuesday, September 10, 2013 11:15 AM**** > > > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > if you have set these values inside hive-site.xml and restarted hive > service. It should work. **** > > **** > > Share your hive-site.xml (please remove values for s3naccess key and > secret ) **** > > **** > > On Tue, Sep 10, 2013 at 3:17 AM, <shouvanik.hal...@accenture.com> wrote:** > ** > > Please help. I am badly stuck on this. Please help.**** > > **** > > *From:* Haldar, Shouvanik > *Sent:* Tuesday, September 10, 2013 1:03 AM**** > > > *To:* 'user@hive.apache.org' > *Subject:* RE: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > When I set values inside hive-site.xml & test whether values are set by > typing hive> set; I see that the values are set.**** > > **** > > Yet, when I run “CREATE EXTERNAL TABLE…” command, I get the same error**** > > **** > > FAILED: Error in metadata: > MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID > and Secret Access Key must be specified as the username or password > (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or > fs.s3n.awsSecretAccessKey properties (respectively).)**** > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask**** > > **** > > Please help.**** > > **** > > Thanks,**** > > Shouvanik**** > > **** > > *From:* Haldar, Shouvanik **** > > *Sent:* Tuesday, September 10, 2013 12:39 AM**** > > *To:* 'user@hive.apache.org' > *Subject:* RE: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > Hi Nitin,**** > > **** > > Please help me with the below?**** > > **** > > Thanks,**** > > Shouvanik**** > > **** > > *From:* Haldar, Shouvanik **** > > *Sent:* Saturday, September 07, 2013 2:30 AM**** > > *To:* user@hive.apache.org > *Subject:* RE: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > Hi ,**** > > **** > > I tried to set the parameters inside hive-site.xml on the node from where > I am running hive. Even “hive>set;” command showed me that the parameters > were successfully set. Yet, when running hive “CREATE EXTERNAL TABLE…”, I > am getting the following error**** > > **** > > FAILED: Error in metadata: > MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID > and Secret Access Key must be specified as the username or password > (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or > fs.s3n.awsSecretAccessKey properties (respectively).)**** > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask **** > > **** > > **** > > Please help.**** > > **** > > Thanks,**** > > Shouvanik**** > > **** > > *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com<nitinpawar...@gmail.com>] > **** > > *Sent:* Saturday, September 07, 2013 2:25 AM**** > > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > did you try setting them in hive-site.xml file ? **** > > **** > > On Sat, Sep 7, 2013 at 2:18 AM, <shouvanik.hal...@accenture.com> wrote:*** > * > > Hi,**** > > **** > > Can anybody please reply to this below? I am stuck up very bad.**** > > **** > > Regards,**** > > Shouvanik**** > > **** > > *From:* Haldar, Shouvanik > *Sent:* Friday, September 06, 2013 11:22 PM > *To:* user@hive.apache.org > *Subject:* RE: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > Hi,**** > > **** > > Is it possible to permanently set the below parameters? Please help.**** > > **** > > Thanks,**** > > SHouvanik**** > > **** > > *From:* Jitendra Yadav > [mailto:jeetuyadav200...@gmail.com<jeetuyadav200...@gmail.com>] > > *Sent:* Thursday, September 05, 2013 6:02 PM**** > > > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > OK got your point, As an alternative you can set these variables in hive > environment before creating the table**** > > **** > > Hive > set fs.s3n.awsSecretAccessKey='value' > Hive > set fs.s3n.awsAccessKeyId='Value'**** > > **** > > Hope this will help you.**** > > **** > > Thanks**** > > Jitendra > On Thu, Sep 5, 2013 at 5:19 PM, <shouvanik.hal...@accenture.com> wrote:*** > * > > Hi,**** > > **** > > Thanks for the reply.**** > > **** > > I had seen in some posts, that you could mention these keys on the Hive > CLI. If things work, then I am later planning to include it in > core-site.xml of Hadoop config. Presently, I am not allowed to restart > Hadoop services. **** > > **** > > “create external table ….**** > > …..**** > > LOCATION 's3n:// <fs.s3n.awsAccessKeyId: > fs.s3n>.<awsSecretAccessKey>@ap-hw-dev/service-names'; > **** > > **** > > Please help me.**** > > **** > > Thanks,**** > > Shouvanik**** > > **** > > *From:* Jitendra Yadav [mailto:jeetuyadav200...@gmail.com] > *Sent:* Thursday, September 05, 2013 5:11 PM > *To:* user@hive.apache.org > *Subject:* Re: error executing the tutorial -> > https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely > **** > > **** > > Hi,**** > > **** > > Did you configured fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey > properties**** > > in your core-site.xml file?**** > > **** > > Regards**** > > Jitendra**** > > On Thu, Sep 5, 2013 at 3:52 PM, <shouvanik.hal...@accenture.com> wrote:*** > * > > Hi,**** > > **** > > I am executing the example, but got initial roadblock while trying out > “Setting up tables (DDL Statements)” section.**** > > **** > > When I executed the command “hive> create external table kv (key int, > values string) location 's3n://data.s3ndemo.hive/kv';”**** > > **** > > I got the error**** > > **** > > FAILED: Error in metadata: > MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID > and Secret Access Key must be specified as the username or password > (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or > fs.s3n.awsSecretAccessKey properties (respectively).)**** > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask**** > > hive>**** > > **** > > Please help.**** > > **** > > **** > > Thanks,**** > > Shouvanik**** > > **** > ------------------------------ > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. > > Where allowed by local law, electronic communications with Accenture and > its affiliates, including e-mail and instant messaging (including content), > may be scanned by our systems for the purposes of information security and > assessment of internal compliance with Accenture policy. > > > ______________________________________________________________________________________ > > www.accenture.com**** > > **** > > **** > > > > **** > > **** > > -- > Nitin Pawar**** > > > > **** > > **** > > -- > Nitin Pawar**** > > > > **** > > **** > > -- > Nitin Pawar**** > > > > **** > > ** ** > > -- > Nitin Pawar**** > -- Nitin Pawar
