Re: Fw: Hive update operation

Hi Dr Mich, Thank you for replying. Yes, while creating the table, transactional property has been set as true, same implies to other tables. Following in are SQL update query example, same I want to achieve through HQL: 1) UPDATE table1 SET FAging=t2.FAging, PaymentR=t2.PaymentR, Arrear

Hive + Kerberos question

I have a question about Hive + Kerberos. Perhaps I'm missing something, perhaps it's an oversight, perhaps it's a bug. I can get a TGT ticket using kinit, but it's easier for me to get one using JAAS since there's no dependency on an external command and I can nuke the keytab file immediately afte

Re: unsubscribe

Sent from my iPhone > On Aug 23, 2016, at 9:53 PM, Mark Memory wrote: > > > >

Re: Loading Sybase to hive using sqoop

If this is one off then Spark will do OK. Sybase IQ provides bcp that creates a flat file tab/comma separated and you can use that to extract IQ table and put it on HDFS and create an external table. This is of course is a one off. You can also use SRS (SAP Replication Server) to get the data ou

Re: Loading Sybase to hive using sqoop

> val d = HiveContext.read.format("jdbc").options( ... >> The sqoop job takes 7 hours to load 15 days of data, even while setting >>the direct load option to 6. Hive is using MR framework. In generaly, the jdbc implementations tend to react rather badly to large extracts like this - the throttli

Sqoop vs spark jdbc

Team, Please help me in choosing sqoop or spark jdbc to fetch data from rdbms. Sqoop has lot of optimizations to fetch data does spark jdbc also has those ?  I'm performing few analytics using spark data for which data is residing  in rdbms. Please guide me with this. Thanks Venkata Karthik P

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

This is also a good option. With respect to Hive transactional tables: I do to think they have been designed for massive inserts of single items. On the other hand you would not insert a lot of events using single inserts in a relational database. Same restrictions apply, it is not the use case

Re: Loading Sybase to hive using sqoop

Is your Sybase server ready to deliver a large amount of data? (Network, memory, cpu, parallel access, resources etc) This is usually the problem when loading data from a relational database and less sqoop / mr or spark. Then, you should have a recent Hive version and store in Orc or parquet

Re: Loading Sybase to hive using sqoop

Are you using a vendor distro or in-house build? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaim

Re: Loading Sybase to hive using sqoop

hm. Watching paint dry :) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own

Re: Loading Sybase to hive using sqoop

We are running hive on MR On Wednesday, August 24, 2016, Mich Talebzadeh wrote: > Sybase IQ uses jconn4.jar for ODBC connection. This is the way I use Spark > to get IQ data into Hive table. You can specify partition in Sqoop as well. > > I started using Sqoop to populate Hive tables but decided

Re: Loading Sybase to hive using sqoop

Sybase IQ uses jconn4.jar for ODBC connection. This is the way I use Spark to get IQ data into Hive table. You can specify partition in Sqoop as well. I started using Sqoop to populate Hive tables but decided to use Spark. Also are you running Hive on Map-reduce engine? private var dbURL = "jd

Re: Fw: Hive update operation

have the underlying table to be updated been defined as transactional? can you give the update example? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

Thanks Kit ! We are trying HDFS bolt with external tables now. On Wed, Aug 24, 2016 at 5:25 PM, Kit Menke wrote: > Joel, > Another option which you have is to use the Storm HDFS bolt to stream data > into Hive external tables. The external tables then get loaded into ORC > history tables for lon

Re: HIVE on Windows

Only with a Hadoop distribution that supports Windows. Generally I recommend to avoid installing single pieces of the ecosystem, but use a proper distribution. Their underlying default configuration is usually better and they usually tested the integration properly. > On 24 Aug 2016, at 15:23,

Fw: Hive update operation

From: Priyanka Raghuvanshi Sent: 24 August 2016 19:12 To: user@hive.apache.org Subject: Hive update operation Hi, Is it possible to update a table(ORC) using join or column of another table(ORC)? Kind Regards, Priyanka Raghuvanshi

Re: HIVE on Windows

Hi, Were you able to get Hive up and running on Windows machine ? I have installed Hadoop on Windows now I want to install Hive too. I couldn't find binaries to run on Windows machine. Can anyone tell me is it possible to run Hive on Windows machine ? Thanks On Wednesday, May 18, 2016, Me To wro

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

Joel, Another option which you have is to use the Storm HDFS bolt to stream data into Hive external tables. The external tables then get loaded into ORC history tables for long term storage. We use this in a HDP cluster with similar load so I know it works. :) I'm with Jörn on this one. My impress

Too many hive client cause mysql meta store hang infinitely

Hi, we encounter this problem in our proc hadoop cluster: Sometimes hive mysql metastore hang because deadlock, so result in all hive client hang infinitely. we check in this problem, found most mysql request hang on " SHOW FULL COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%' " More

Re: How to remove Hive table property?

Creating new table with subsequent rename seems good for me. Thanks for advice, Mich. On Tue, Aug 23, 2016 at 3:12 PM, Mich Talebzadeh wrote: > Has table got data in it? > > Can you create a new table WITHOUT serialization.null.format and > INSERT/SELECT from old to new, drop old and rename new

Re: Hive transaction doesn't release lock.

Oh, my copy paste skills failed me. Here's output from HIVE_LOCKS: # HL_LOCK_EXT_ID, HL_LOCK_INT_ID, HL_TXNID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE, HL_LOCK_TYPE, HL_LAST_HEARTBEAT, HL_ACQUIRED_AT, HL_USER, HL_HOST '8496355', '1', '78461824', 'default', 'data_http', 'dt=20160821', 'a', 'r'

Apache Hive 2 - Spark 2 Integration, Failed to create spark client Exception

Hi, I want to set execution engine for Hive as Spark. I deployed a 4 server Spark cluster in Yarn mode. When Ever am trying to execute any query am getting the following error. Please suggest me weather I done something wrong . Also attaching the used tools version. Hive 2.1.0 , HBase 1.2.2

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

@Jörn: If I understood correctly even later versions of Hive won't be able to handle these kinds of workloads? On Wed, Aug 24, 2016 at 1:26 PM, Jörn Franke wrote: > I think Hive especially these old versions have not been designed for > this. Why not store them in Hbase and run a oozie job regul

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

I think Hive especially these old versions have not been designed for this. Why not store them in Hbase and run a oozie job regularly that puts them all into Hive /Orc or parquet in a bulk job? > On 24 Aug 2016, at 09:35, Joel Victor wrote: > > Currently I am using Apache Hive 0.14 that ships

Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

Currently I am using Apache Hive 0.14 that ships with HDP 2.2. We are trying perform streaming ingestion with it. We are using the Storm Hive bolt and we have 7 tables in which we are trying to insert. The RPS (requests per second) of our bolts ranges from 7000 to 5000 and our commit policies are c