Thanks Nitin for your help.I was able to run hive and also start meta store finally !! The tar should contain required config files(instead of having templates), otherwise it turns out to be a difficult for newbies like me to go ahead and learn/use hive. Also I could not find the port number for metastore and had to hardcode it in hcat_server.sh.
On Mon, Oct 7, 2013 at 3:29 PM, Nitin Pawar <nitinpawar...@gmail.com> wrote: > So here is what I did quickly. (I did not follow the step by step guide > from hive wiki) > 1) I already have a running hadoop cluster > 2) Downloaded the bin-tar.gz from hive.apache.org links > 3) Uncompressed the binaries > 4) Inside the conf folder there are few .template files (I don't know > there is no hive-site.xml there. In core hadoop projects the core-site.xml > is empty but its present) > 5) I just copied all the template files non template files (eg. > hive-defaults.xml.template to hive-defaults.xml and so on) > 6) I created a a new hive-site.xml file > 7) added following properties to it (which are kind of missing from > hive-defaults but we will talk on dev forums for that) > > - javax.jdo.option.ConnectionURL - I use mysql as my meta store so i > have pointed this to > jdbc:mysql://<myhost>:<myport>/<mydbname>?createDatabaseIfNotExist=true > - javax.jdo.option.ConnectionDriverName - this is set to > com.mysql.jdbc.driver I have downloaded mysql driver from oracle and > stored in hive lib folder. > - javax.jdo.option.ConnectionUserName - this is my db username > - javax.jdo.option.ConnectionPassword - this is password for the > username i used above > > 8) export HADOOP_HOME variable to where my hadoop installation is present > 9) export HIVE_HOME to the directory where I have extracted the tar ball > 10) Granted access to new machine on mysql db > 11) started hive metastore and verified show tables. > > This worked. Of course this does not take care of security on hive yet. > > if run into any issues after this do let me know. > > > On Sat, Oct 5, 2013 at 7:50 PM, Nitin Pawar <nitinpawar...@gmail.com>wrote: > >> Baahu, >> >> hive has two conf files. One is hive-site.xml and other is hive-env.sh >> >> From hive-0.11 hcat is bundled with hive so you specially need not do >> anything special to setup hcat. >> you just need to setup few environment variables like HIVE_HOME, >> HIVE_CONF_DIR, HADOOP_HOME. >> >> and then if you have setup the hive-site.xml correctly then it should >> work out of the box. >> >> If you have downloaded the already built tarball. you need not compile >> it. Plus its really a long build if you run it with unit test cases etc. >> >> If possible tonight I will setup a vm and test out the steps mentioned >> on hive wiki. >> >> >> On Sat, Oct 5, 2013 at 7:21 PM, Baahu <bahub...@gmail.com> wrote: >> >>> Mikhail, >>> The hive-site.xml was missing and while googling I found some suggestion >>> of copying hive-0.11.0-bin/hcatalog/etc/hcatalog/proto-hive-site.xml as >>> hive-site.xml to the directory hive-0.11.0-bin/conf . >>> Was this a right instruction, if not can you please let me know where I >>> can find hive-conf.xml file. >>> >>> Should we really have to build hive as mentioned in the below link >>> during installation? Just untarring the hive tar ball doesn't lead to its >>> installation? >>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted >>> >>> Thanks, >>> Baahu >>> >>> >>> On Sat, Oct 5, 2013 at 3:26 PM, Mikhail Antonov >>> <olorinb...@gmail.com>wrote: >>> >>>> Bases on original question I decided the author does not need hcatalog >>>> specifically, but got sample hive-site.xml suggesting storage authenticator >>>> somewhere. So why use it?:) >>>> >>>> On Oct 5, 2013, at 2:51, Lefty Leverenz <leftylever...@gmail.com> >>>> wrote: >>>> >>>> P.S. I found StorageDelegationAuthorizationProvider mentioned in the >>>> Known Issues section of the HCatalog authorization doc: "A different >>>> authorization provider (StorageDelegationAuthorizationProvider) needs to be >>>> used for working with HBase tables as well. But that is not well tested >>>> ." >>>> >>>> -- Lefty >>>> >>>> >>>> On Sat, Oct 5, 2013 at 2:44 AM, Lefty Leverenz <leftylever...@gmail.com >>>> > wrote: >>>> >>>>> I'm out of my depth, but based on existing documentation I'll make >>>>> some suggestions you could try while waiting for replies from people who >>>>> actually know about installation. >>>>> >>>>> Following Mikhail's lead, since this is an HCatalog security problem >>>>> you should start by seeing if HCatalog is installed. Then you could try >>>>> changing the value of hive.security.authorization.manager -- I found it >>>>> documented in two places: >>>>> >>>>> - Hive Configuration >>>>> Properties<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Authentication%2FAuthorization> >>>>> - HCatalog Storage Based >>>>> Authorization<https://cwiki.apache.org/confluence/display/Hive/HCatalog+Authorization#HCatalogAuthorization-ConfiguringStorageSystemBasedAuthorization> >>>>> >>>>> The HCatalog authorization doc says you need to add the HCatalog jar >>>>> to the Hive classpath (add "export HIVE_AUX_JARS_PATH=<*path to >>>>> hcatalog jar*>" to hive-env.sh), but I'm not sure if that's needed >>>>> now that HCatalog has been integrated with Hive. >>>>> >>>>> Try changing hive.security.authorization.manager to its default value, >>>>> either the one listed in the Hive doc or the one in the HCatalog doc: >>>>> >>>>> - >>>>> >>>>> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider >>>>> - org.apache.hcatalog.security.HdfsAuthorizationProvider >>>>> >>>>> Or you could turn off hive.security.authorization.enabled by setting >>>>> it to false (see the HCatalog authorization doc). There's more >>>>> information >>>>> about this property in the Hive Authorization doc's >>>>> Prerequisites<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization#LanguageManualAuthorization-Prerequisites>section. >>>>> >>>>> Changing configuration property values is documented in Getting >>>>> Started (Configuration management >>>>> overview<https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Configurationmanagementoverview>) >>>>> as well as Configuring >>>>> Hive<https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-ConfiguringHive> >>>>> . >>>>> >>>>> Agreed, the documentation is confusing. We're trying to improve it. >>>>> >>>>> -- Lefty >>>>> >>>>> >>>>> On Sat, Oct 5, 2013 at 12:53 AM, Baahu <bahub...@gmail.com> wrote: >>>>> >>>>>> Lefty, >>>>>> Can the install instructions given at >>>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted be >>>>>> followed as is? >>>>>> I see instructions to build from source code just after "Installing >>>>>> Hive from a Stable Release" section, which is kinda confusing.I bump in >>>>>> to >>>>>> the error mentioned above while I execute the instructions in "Running >>>>>> Hive" section. >>>>>> >>>>>> Thanks, >>>>>> Baahu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Oct 5, 2013 at 1:17 PM, Lefty Leverenz < >>>>>> leftylever...@gmail.com> wrote: >>>>>> >>>>>>> Do you have any suggestions for improving the Hive wiki instructions? >>>>>>> >>>>>>> -- Lefty >>>>>>> >>>>>>> >>>>>>> On Fri, Oct 4, 2013 at 7:30 PM, Baahu <bahub...@gmail.com> wrote: >>>>>>> >>>>>>>> Apache hive. >>>>>>>> On Oct 5, 2013 1:06 AM, "Mikhail Antonov" <olorinb...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you using apache hive or one from cloudera/Horton works? >>>>>>>>> >>>>>>>>> On Oct 4, 2013, at 11:52, Baahu <bahub...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi Mikhail, >>>>>>>>> I followed the steps mentioned in the Apache hive website, found >>>>>>>>> the instructions not so helpful... Is there any good link which >>>>>>>>> provides >>>>>>>>> proper installation steps. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> On Oct 5, 2013 12:07 AM, "Mikhail Antonov" <olorinb...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Looks like you have StorageDelegationAuthorizationProvider >>>>>>>>>> enabled in hive-site.xml (like below), but hcatalog isn't installed >>>>>>>>>> somehow? >>>>>>>>>> >>>>>>>>>> <property> >>>>>>>>>> <name>hive.security.authorization.manager</name> >>>>>>>>>> >>>>>>>>>> <value>org.apache.hcatalog.security.StorageDelegationAuthorizationProvider</value> >>>>>>>>>> <description>the hive client authorization manager class name. >>>>>>>>>> The user defined authorization class should implement interface >>>>>>>>>> org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider. >>>>>>>>>> HCatalog uses a model, where authorization checks are delegated to >>>>>>>>>> the storage layer (hdfs, hbase, ...). >>>>>>>>>> </description> >>>>>>>>>> </property> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013/10/4 Baahu <bahub...@gmail.com> >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Have installed Hive 0.11 in centos.I see the below error when I >>>>>>>>>>> execute hive.Can you please help me resolving this. >>>>>>>>>>> >>>>>>>>>>> [root@centosvm bin]# hive >>>>>>>>>>> 13/10/04 09:05:35 WARN conf.HiveConf: DEPRECATED: Configuration >>>>>>>>>>> property hive.metastore.local no longer has any effect. Make sure to >>>>>>>>>>> provide a valid value for hive.metastore.uris if you are connecting >>>>>>>>>>> to a >>>>>>>>>>> remote metastore. >>>>>>>>>>> >>>>>>>>>>> Logging initialized using configuration in >>>>>>>>>>> jar:file:/opt/hive-0.11.0-bin/lib/hive-common-0.11.0.jar!/hive-log4j.properties >>>>>>>>>>> Hive history >>>>>>>>>>> file=/tmp/root/hive_job_log_root_8553@centosvm_201310040905_1657489660.txt >>>>>>>>>>> Exception in thread "main" java.lang.RuntimeException: >>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveException: >>>>>>>>>>> java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) >>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>>>> Method) >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:160) >>>>>>>>>>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >>>>>>>>>>> java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:342) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:272) >>>>>>>>>>> ... 7 more >>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider >>>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>> Method) >>>>>>>>>>> at >>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>> at java.lang.Class.forName(Class.java:270) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:335) >>>>>>>>>>> ... 8 more >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks, >>>>>>>>>> Michael Antonov >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Twitter:http://twitter.com/Baahu >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Twitter:http://twitter.com/Baahu >>> >>> >> >> >> -- >> Nitin Pawar >> > > > > -- > Nitin Pawar > -- Twitter:http://twitter.com/Baahu
