Re: LINES TERMINATED BY only supports newline '\n' right now

Did you try defining the table with hive In built SerDe. 'Stored as ORC' This should resolve your issue. Plz try and let me know if it works. Abhi Sent from my iPhone > On Jun 3, 2016, at 3:33 AM, Markovitz, Dudu wrote: > > Here is an example, but first – some warnings: > > · You s

Re: Creating a Hive table through Spark and potential locking issue (a bug)

Hi, Just to clarify I use Hive with Spark engine (default) so Hive on Spark engine as we discussed and observed. Now with regard to Spark (as an app NOT execution engine) doing the create table in Hive and populating it, I don't think Spark itself does any transactional enforcement. This means th

Re: Creating a Hive table through Spark and potential locking issue (a bug)

Locks in Hive are acquired by the query complier and should be independent of the execution engine. Having said that, I've not tried this on Spark, so my answer is only accurate with Hive. Eugene From: Michael Segel mailto:msegel_had...@hotmail.com>> Reply-To: "user@hive.apache.org

Re: Creating a Hive table through Spark and potential locking issue (a bug)

yes, I assumed that you had Acid enabled. I don’t think what I said would be true w/o this. From: Mich Talebzadeh mailto:mich.talebza...@gmail.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Wednesday, June 8, 2016 at 4:06 PM To: user mai

Re: NPE while querying timestamp data type in ORC tables

Hi This is a known issue and is fixed as part of https://issues.apache.org/jira/browse/HIVE-10592 Workaround is to set hive.optimize.index.filter=false but this will disable predicate pushdown and will have performance impact. This issue can happen when all the values in the timestamp column pe

NPE while querying timestamp data type in ORC tables

Hi all, I have a ORC Hive table with multiple columns. One of the column data type (say column1) is of type timestamp. When I make a query 'select * from table_name where column1 is null;', I get Null Pointer Exception. Has anyone faced similar issue when you are making similar queries on timesta

Re: Creating a Hive table through Spark and potential locking issue (a bug)

BTW DbTxnManager is set as well hive.txn.manager *org.apache.hadoop.hive.ql.lockmgr.DbTxnManager* Set to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager as part of turning on Hive transactions, which also requires appropriate settings for hive.compactor.initiator.on, hi

Re: Creating a Hive table through Spark and potential locking issue (a bug)

OK this seems to work. 1. Create the target table first 2. Populate afterwards I first created the target table with hive> create table test.dummy as select * from oraclehadoop.dummy where 1 = 2; Then did INSERT/SELECT and tried to drop the target table when DML (INSERT/SELECT) was g

Re: Creating a Hive table through Spark and potential locking issue (a bug)

> On Jun 8, 2016, at 3:35 PM, Eugene Koifman wrote: > > if you split “create table test.dummy as select * from oraclehadoop.dummy;” > into create table statement, followed by insert into test.dummy as select… > you should see the behavior you expect with Hive. > Drop statement will block while

Re: Creating a Hive table through Spark and potential locking issue (a bug)

if you split “create table test.dummy as select * from oraclehadoop.dummy;” into create table statement, followed by insert into test.dummy as select… you should see the behavior you expect with Hive. Drop statement will block while insert is running. Eugene From: Mich Talebzadeh mailto:mich.ta

Re: Creating a Hive table through Spark and potential locking issue (a bug)

Hive version is 2 We can discuss all sorts of scenarios. However, Hivek is pretty good at applying the locks at both the table and partition level. The idea of having a metadata is to enforce these rules. [image: Inline images 1] For example above inserting from source to target table partition

Re: Creating a Hive table through Spark and potential locking issue (a bug)

Doh! It would help if I use the email address to send to the list… Hi, Lets take a step back… Which version of Hive? Hive recently added transaction support so you have to know your isolation level. Also are you running spark as your execution engine, or are you talking about a spark a

Re: Creating a Hive table through Spark and potential locking issue (a bug)

Hi, The idea of accessing Hive metada is to be aware of concurrency. In generall if I do the following In Hive hive> create table test.dummy as select * from oraclehadoop.dummy; We can see that hive applies the locks in Hive [image: Inline images 2] However, there seems to be an

Re: JsonSerDe file format question

g.apache.hive.hcatalog.data.JsonSerDe' > location '/mytest/test'; > > and added partition > alter table testtable > add if not exists partition (mmdd=20160608) location > '/mytest/test/20160608'; > > > There are 3 file with r JSON records each. But when I run select * from > testtable; it return me only first row from each one of file nested of 9. > > What can be the problem? >

Where are jars stored for permanent functions

Hi All, I just added local jars to my hive session, created permanent functions, and find that they are available across sessions and machines. This is of course excellent, but I'm wondering where those jars are being stored? What setting or what default directory would I find them in. My session

RE: Creating a Hive table through Spark and potential locking issue (a bug)

Could you be looking at 2 jobs trying to use the same file and one getting to it before the other and finally removing it? David Newberger From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] Sent: Wednesday, June 8, 2016 1:33 PM To: user; user @spark Subject: Creating a Hive table through S

Re: JsonSerDe file format question

t; url string, > ...) > partitioned by (mmdd int) > ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' > location '/mytest/test'; > > and added partition > alter table testtable > add if not exists partition (mmdd=20160608) location > '

JsonSerDe file format question

testtable ( id bigint, url string, ...) partitioned by (mmdd int) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' location '/mytest/test'; and added partition alter table testtable add if not exists partition (mmdd=20160608) location '/mytest/test/20160608';

Creating a Hive table through Spark and potential locking issue (a bug)

Hi, I noticed an issue with Spark creating and populating a Hive table. The process as I see is as follows: 1. Spark creates the Hive table. In this case an ORC table in a Hive Database 2. Spark uses JDBC connection to get data out from an Oracle 3. I create a temp table in Spark th

Re: Delete hive partition while executing query.

This looks like proof of a bug. The reads locks 179730 and 179731 should have been blocked by 179729. As Alan said this won’t prevent the exception you are getting but it needs to be fixed to prevent a partition from disappearing while query 3 and 4 are in progress. Could you file a Jira pleas

RE: Need Your Inputs For Below Scenario

Here Table2 is very large table and contains lakhs of rows. From: Lunagariya, Dhaval [CCC-OT] Sent: Wednesday, June 08, 2016 5:52 PM To: user@hive.apache.org Subject: Need Your Inputs For Below Scenario Hey folks, Need your help. Input Table1: Column1 Column2 Column3 Column4 Column5 Colum

Re: Delete hive partition while executing query.

Hi, thanks for reply, Alan. Here's one more test. Wed Jun 08 16:36:02 MSK 2016 Start thread 1 Wed Jun 08 16:36:05 MSK 2016 Start thread 2 Wed Jun 08 16:36:08 MSK 2016 Start thread 3 Wed Jun 08 16:36:11 MSK 2016 Start thread 4 Wed Jun 08 16:36:17 MSK 2016 Finish thread 1 Wed Jun 08 16:36:17 MSK 201

Need Your Inputs For Below Scenario

Hey folks, Need your help. Input Table1: Column1 Column2 Column3 Column4 Column5 Column6 Column7 A B1 B2 B3(NULL) B4 B5 B6 Input Table2: Column1 Column2 B1 D1 B2 D2 B3 D3 B4 D4 B5 D5 B6 D6 Output: Column1 Column2 Column3 A B1 D1 A B2 D2 A B4 D4 A B5 D

Re: insert query in hive

This is not the recommended way to load large data volumes into Hive. Check the external table feature, scoop, and the Orc/parquet formats > On 08 Jun 2016, at 14:03, raj hive wrote: > > Hi Friends, > > I have to insert the data into hive table from Java program. Insert query > will work in

insert query in hive

Hi Friends, I have to insert the data into hive table from Java program. Insert query will work in Hive directly? like below sql command insert into tablename values(value1,value2) Thanks Raj

Re: alter partitions on hive external table

Thank you so much for the detailed info. On Tue, Jun 7, 2016 at 1:58 AM, Markovitz, Dudu wrote: > And here is a full example > > > > > > > -- bash > > > --