Abshiek What is the version of Sqoop you are using? Also can you paste in the sqoop command you use with the full stack trace? Make sure that you have the required jdbc driver jar in the /lib directory of SQOOP.
Regards Bejoy ________________________________ From: Abhishek Parolkar <abhis...@viki.com> To: user@hive.apache.org Sent: Friday, March 30, 2012 1:33 PM Subject: Re: Postgres JDBC + dboutput UDF to export from Hive to remote Postgres forgot to add the sceenshot in last email :) On Fri, Mar 30, 2012 at 4:02 PM, Abhishek Parolkar <abhis...@viki.com> wrote: I even tried sqoop, but with no luck. It complains for connection manager even though my mysql connector jar is in lib folder of sqoop's instalation dir. > > >Any help? > > >If sqoop's purpose is to allow import/export from RDBMS, why the basic >mysql/pg connectors bundled with it? > > >-v_abhi_v > > > >On Fri, Mar 30, 2012 at 9:50 AM, Abhishek Parolkar <abhis...@viki.com> wrote: > >I am definitely getting the no-driver error http://screencast.com/t/OipV14n9FgF >>so its not even at the point of executing statements, my return value from >>UDF is 2. >> >> >>I can confirm that the postgres driver jar is added to my hadoop_classpath >>and also to get this working I am working on 1 node local cluster. >> >> >>-v_abhi_v >> >> >> >>On Thu, Mar 29, 2012 at 10:11 PM, Edward Capriolo <edlinuxg...@gmail.com> >>wrote: >> >>You have to look at the code to see what the return numbers mean for >>>the UDF. In some cases the return is normal. For example hive maps use >>>speculative execution and the same insert happens twice violating a >>>primary key. The second insert "fails" and produces non 0 but in >>>reality all that means is already inserted/ >>> >>> >>>On Thu, Mar 29, 2012 at 6:23 AM, Abhishek Parolkar <abhis...@viki.com> wrote: >>>> My situation requires me to run hive query every hour and insert selected >>>> records to postgres table. It would be nice if dboutput works so that >>>> reduce >>>> jobs (created by hive) can directly write to DB. >>>> >>>> With sqoop, I will have to create a table everytime in hive and export it >>>> to >>>> a table in DB. Wondering if that can be avoided? >>>> >>>> -v_abhi_v >>>> >>>> >>>> On Thu, Mar 29, 2012 at 6:12 PM, Bejoy KS <bejoy...@yahoo.com> wrote: >>>>> >>>>> Hi Abshiek >>>>> To transfer data between rdbms and hadoop Sqoop is the preferred and >>>>> recommended option. Once you have the process done in hive the output data >>>>> can be exported to PG with sqoop export command. >>>>> Regards >>>>> Bejoy KS >>>>> >>>>> Sent from handheld, please excuse typos. >>>>> ________________________________ >>>>> From: Abhishek Parolkar <abhis...@viki.com> >>>>> Date: Thu, 29 Mar 2012 16:25:08 +0800 >>>>> To: <user@hive.apache.org> >>>>> ReplyTo: user@hive.apache.org >>>>> Subject: Postgres JDBC + dboutput UDF to export from Hive to remote >>>>> Postgres >>>>> >>>>> Hi There, >>>>> I am trying to get dboutput() UDF to work so that it can write result to >>>>> a PG DB table. >>>>> >>>>> ==This is what I did in hive shell== >>>>> >>>>> add jar /location/hive_contrib.jar; >>>>> add jar /location/postgresql9jdbc3.jar; >>>>> set jdbc.drivers = org.postgresql.Driver; >>>>> >>>>> CREATE TEMPORARY FUNCTION dboutput >>>>> AS 'org.apache.hadoop.hive.contrib.genericudf.example.GenericUDFDBOutput'; >>>>> >>>>> select dboutput('jdbc:postgresql//localhost:5432/test','','','insert >>>>> into test_tbl(cnt) values(?)',hex(count(*))) >>>>> from some_hive_table >>>>> >>>>> ===========end of snip======= >>>>> >>>>> 1.) I am on single node cluster >>>>> 2.) I am using Hive 0.8.1 >>>>> 3.) I on hadoop 1.0.0 >>>>> 4.) query runs fine but doesnt write to DB, it returns number 2 >>>>> (http://screencast.com/t/eavnbBHR1x) >>>>> >>>>> I get no suitable driver error (http://screencast.com/t/OipV14n9FgF) , can >>>>> some one tell me how can I load postgres JDBC such >>>>> that dboutput recognizes my postgres. >>>>> >>>>> Any help? >>>>> >>>>> -v_abhi_v >>>> >>>> >>> >> >
