Re: Loading Sybase to hive using sqoop

2016-08-25 Thread Rahul Channe
Thank you all for valuable inputs On Wednesday, August 24, 2016, Mich Talebzadeh wrote: > If this is one off then Spark will do OK. > > Sybase IQ provides bcp that creates a flat file tab/comma separated and > you can use that to extract IQ table and put it on HDFS and create an > external table

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Mich Talebzadeh
If this is one off then Spark will do OK. Sybase IQ provides bcp that creates a flat file tab/comma separated and you can use that to extract IQ table and put it on HDFS and create an external table. This is of course is a one off. You can also use SRS (SAP Replication Server) to get the data ou

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Gopal Vijayaraghavan
> val d = HiveContext.read.format("jdbc").options( ... >> The sqoop job takes 7 hours to load 15 days of data, even while setting >>the direct load option to 6. Hive is using MR framework. In generaly, the jdbc implementations tend to react rather badly to large extracts like this - the throttli

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Jörn Franke
Is your Sybase server ready to deliver a large amount of data? (Network, memory, cpu, parallel access, resources etc) This is usually the problem when loading data from a relational database and less sqoop / mr or spark. Then, you should have a recent Hive version and store in Orc or parquet

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Mich Talebzadeh
Are you using a vendor distro or in-house build? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaim

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Mich Talebzadeh
hm. Watching paint dry :) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Rahul Channe
We are running hive on MR On Wednesday, August 24, 2016, Mich Talebzadeh wrote: > Sybase IQ uses jconn4.jar for ODBC connection. This is the way I use Spark > to get IQ data into Hive table. You can specify partition in Sqoop as well. > > I started using Sqoop to populate Hive tables but decided

Re: Loading Sybase to hive using sqoop

2016-08-24 Thread Mich Talebzadeh
Sybase IQ uses jconn4.jar for ODBC connection. This is the way I use Spark to get IQ data into Hive table. You can specify partition in Sqoop as well. I started using Sqoop to populate Hive tables but decided to use Spark. Also are you running Hive on Map-reduce engine? private var dbURL = "jd