Re: Hive Compaction OOM

hive.compactor.max.num.delta This lets control how many deltas are opened at once. By default it’s 500 which may be too much. So the compactor will use this do exactly what Owen is suggesting. The current impl will do everything sequentially but better than OOM. Eugene From: Owen O'Malley Re

Re: Does MERGE feature works only on TEZ in Hive-2.3?

It works on MR as well. From: Oleksiy S Reply-To: "user@hive.apache.org" Date: Wednesday, April 18, 2018 at 5:24 AM To: "user@hive.apache.org" , "d...@hive.apache.org" Subject: Does MERGE feature works only on TEZ in Hive-2.3? Hi all! Does MERGE feature works only on TEZ in Hive-2.3? see

Re: ORC ACID table returning Array Index Out of Bounds

What version of Hive is this? Can you isolate this to a specific partition? The table/partition you are reading should have a directory called base_x/ with several bucket_N files. (if you see more than 1 base_x, take one with highest x) Each bucket_N should have a “hive.acid.key.index

Re: Hive beeline and ACID tables.

“This command is not allowed on an ACID table analytics_user_vws.liveramp_events with a non-ACID transaction manager.” Indicates that “hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager” is not taking effect. When enabling acid you usually want to set this (and other relevant pro

Re: Hive locking mechanism on read partition.

rtition. Hi, Eugene. Tables are not transactional and locks are backed by DbTxnManager. On Fri, Oct 13, 2017 at 2:30 AM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: Which lock manager are you using? Do you have acid enabled and if so are these tables transactional? Eugene From:

Re: Hive locking mechanism on read partition.

Which lock manager are you using? Do you have acid enabled and if so are these tables transactional? Eugene From: Igor Kuzmenko Reply-To: "user@hive.apache.org" Date: Thursday, October 12, 2017 at 3:58 AM To: "user@hive.apache.org" Subject: Hive locking mechanism on read partition. Hello, I'

Re: ORC Transaction Table - Spark

le - Spark Hi, Yes it caused by wrong naming convention of the delta directory : /apps/hive/warehouse/foo.db/bar/year=2017/month=5/delta_0645253_0645253_0001 How do I solve this ? Thanks ! Aviral Agarwal On Tue, Aug 22, 2017 at 11:50 PM, Eugene Koifman mailto:ekoif...@hortonworks.com&g

Re: ORC Transaction Table - Spark

Could you do recursive “ls” in your table or partition that you are trying to read? Most likely you have files that don’t follow expected naming convention Eugene From: Aviral Agarwal Reply-To: "user@hive.apache.org" Date: Tuesday, August 22, 2017 at 5:39 AM To: "user@hive.apache.org" Subjec

Re: Non-local session path expected to be non-null trying to write on Hive using storm-hive

for scheme: hdfs which makes me think I am bundling the wrong HDFS jar in the jar application I'm building. Still, the version being bundled is hdfs 2.6.1, while the version on the cluster is 2.7.3.2.5.5.0-157 (using HDP 2.5) which shouldn't they be compatible? Any suggestion? 2017-

Re: Non-local session path expected to be non-null trying to write on Hive using storm-hive

Are you able to write to Hive to an existing partition? (The stack trace shows that it’s being created) From: Federico D'Ambrosio Reply-To: "d...@hive.apache.org" Date: Monday, July 10, 2017 at 7:38 AM To: "user@hive.apache.org" , "d...@hive.apache.org" Subject: Non-local session path expec

Re: Storm hive bolt

It maybe because you are mixing artifacts from HDP/F and Apache when compiling the topology. Can you try using http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-hive/1.0.1.2.0.1.0-12/ Rather than  org.apache.storm storm-hive 1.0.3 Eugene On 3/29/17, 9:47 AM, "Ma

Re: Hive TxnHandler::lock method run into dead lock.

ed maxConnections up to 50 and recompiled metastore jar. Its gonna be enough for a while. Eugene, do you know is there any Jira item on this problem? On Sun, Mar 26, 2017 at 7:21 PM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: I see (in HikariCP-2.5.1.jar) so perhaps upgradin

Re: Storm hive bolt

https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html has 2016-10-03 23:40:24,322 ERROR [pool-5-thread-114]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(195)) - java.lang.IllegalStateException: Unexpected DataOperationType

Re: Hive TxnHandler::lock method run into dead lock.

ricsTracker"; customizer = new IConnectionCustomizer() { @Override public void customize(Connection connection) throws SQLException { } }; } On Thu, Mar 23, 2017 at 8:29 PM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: Can you try use “hikaricp” connect

Re: Hive TxnHandler::lock method run into dead lock.

Can you try use “hikaricp” connection pool manager? It seems to be using default which is no limit. Eugene From: Igor Kuzmenko Reply-To: "user@hive.apache.org" Date: Monday, March 20, 2017 at 2:17 PM To: "user@hive.apache.org" Subject: Re: Hive TxnHandler::lock method run into dead lock. S

Re: Compactions doesn't launch on Hive 1.1.0

As Gopal said, you have to run a standalone metastore process for compactor to work. More on that in https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreServer. Please note that the transaction specific configuration properties in

Re: Metastore Schema Upgrade : 'ALTER TABLE' cannot be performed on 'COMPACTION_QUEUE' because it does not exist

This table is defined in hive-txn-schema-2.1.0.derby.sql From: sanjeev Reply-To: "user@hive.apache.org" Date: Saturday, January 7, 2017 at 8:12 AM To: "user@hive.apache.org" Subject: Metastore Schema Upgrade : 'ALTER TABLE' cannot be performed on 'COMPACTION_QUEUE' because it does not exist

Re: Problems with Hive Streaming. Compactions not working. Out of memory errors.

The OOM is most likely a side effect of not running compactions. W/o compactions you never reduce the number of delta file that need to be loaded to materialize the data set on read. On 11/29/16, 10:03 AM, "Alan Gates" wrote: >I¹m guessing that this is an issue in the metastore database where it

Re: hive transactional table compaction fails

can you check if the user that the metastore is running as has right to write to the table dir? On 10/26/16, 12:23 PM, "aft" wrote: >On Thu, Oct 27, 2016 at 12:00 AM, Eugene Koifman > wrote: >> could you provide output of ³SHOW COMPACTIONS¹ command > >I have droppe

Re: Error with flush_length File in Orc, in hive 2.1.0 and mr execution engine.

r join attribute a on a.id<http://a.id/> = ml.standard_status Eugene From: satyajit vegesna mailto:satyajit.apas...@gmail.com>> Date: Wednesday, October 26, 2016 at 2:14 PM To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>>, Eugene

Re: hive transactional table compaction fails

gt;Both manual/auto compactor fails. > >On Wed, Oct 26, 2016 at 12:52 AM, Eugene Koifman > wrote: >> does this happen for 1 specific partition or all of them? >> >> On 10/25/16, 12:47 AM, "aft" wrote: >> >>>Hi, >>> >>>Table crea

Re: Error with flush_length File in Orc, in hive 2.1.0 and mr execution engine.

Which of your tables are are transactional? Can you provide the DDL? I don’t think “File does not exist” error is causing your queries to fail. It’s an INFO level msg. There should be some other error. Eugene From: satyajit vegesna mailto:satyajit.apas...@gmail.com>> Reply-To: "user@hive.ap

Re: hive transactional table compaction fails

does this happen for 1 specific partition or all of them? On 10/25/16, 12:47 AM, "aft" wrote: >Hi, > >Table created with this : > >$hive>create table syslog_staged (id string, facility string, >sender string, severity string, tstamp string, service string, msg >string) partitioned by (hostna

Re: Update operation on hive table using Join

This is not supported. The same can be achieved using Merge statement which is WIP: https://issues.apache.org/jira/browse/HIVE-10924. Eugene From: Priyanka Raghuvanshi mailto:priyan...@winjit.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date

Re: populating Hive table periodically from files on HDFS

he author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 26 September 2016 at 04:50, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: Have you considered Hive Streaming? (https://cwiki.apache.org/confluence/display/Hive/Streaming

Re: populating Hive table periodically from files on HDFS

Have you considered Hive Streaming? (https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest) It's built for exactly such use case. Both Flume and Storm are integrated with it and write directly to your target table. Eugene From: Mich Talebzadeh mailto:mich.talebza...@gmail.com

Re: Reading hive-site.xml

HiveConf.java From: Amit Bajpai mailto:amit.baj...@flextronics.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Sunday, September 18, 2016 at 4:57 PM To: "user@hive.apache.org" mailto:user@hive.apache.org>> Subj

Re: Hive transaction doesn't release lock.

:checkLock(1696)) - lockid:8496355 intLockId:1 txnid:78461824 db:default table:data_http partition:dt=20160821 state:WAITING type:SHARED_READ So I guess, that it's race condition between heartbeat thread and TxnReaper thread. Last heartbeat information in HIVE_LOCKS table differs f

Re: Concurrency support of Apache Hive for streaming data ingest at 7K RPS into multiple tables

Hive streaming API (which is what Storm uses) inserts multiple evens to a table per transaction. It has been designed for this but not quite ready for prime time in 0.14. Hive 1.3 has these metastore issues fixed as well as many others. HIVE-11948

Re: Hive transaction doesn't release lock.

your query is "select * from HIVE_LOCKS" but the output is not from HIVE_LOCKS. What entries do you have in HIVE_LOCKS for this txn_id? If all you see is an entry in TXN table in 'a' state - that is OK. that just mean that this transaction was aborted. Eugene From: Igor Kuzmenko mailto:f1she..

Re: Hive compaction didn't launch

torm will get another transaction batch and will wait new messages, which may not come for a long time. I don't see any way to fix this problem with proper configuration, I need to make changes in Hive or Storm code. Question is where it more appropriate? On Fri, Jul 29, 2016 at 8:15 A

Re: Hive compaction didn't launch

I think Storm has some timeout parameter that will close the transaction if there are no events for a certain amount of time. How many transactions do you per transaction batch? Perhaps making the batches smaller will make them close sooner. Eugene On 7/28/16, 3:59 PM, "Alan Gates" wrote: >Bu

Re: Hive sink for Flume, hcatalog error: 'transactional' property is not set on Table

Perhaps you've already seen this https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest and https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions In order to configure Hive properly to stream dat

Re: Optimize Hive Query

if you have many acid tables you almost certainly want more than 2 workers. If you have 2 workers (and a single metastore instance) you can run at most 2 compaction jobs at a time. Unless the tables are very small, compaction may fall behind if it's configured to run too serially. In order fo

Re: Creating a Hive table through Spark and potential locking issue (a bug)

t;, "user @spark" mailto:u...@spark.apache.org>> Subject: Re: Creating a Hive table through Spark and potential locking issue (a bug) On Jun 8, 2016, at 3:35 PM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: if you split "create table test.dummy as select * f

Re: Creating a Hive table through Spark and potential locking issue (a bug)

LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/> On 8 June 2016 at 23:35, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: if you split “create table test.dummy as select * fr

Re: Creating a Hive table through Spark and potential locking issue (a bug)

if you split “create table test.dummy as select * from oraclehadoop.dummy;” into create table statement, followed by insert into test.dummy as select… you should see the behavior you expect with Hive. Drop statement will block while insert is running. Eugene From: Mich Talebzadeh mailto:mich.ta

Re: Delete hive partition while executing query.

-91]: lockmgr.DbLockManager (DbLockManager.java:unlock(182)) - Unlocking lockid:179731 Line 5470: 2016-06-08 16:36:28,131 DEBUG [HiveServer2-Background-Pool: Thread-91]: lockmgr.DbLockManager (DbLockManager.java:unlock(185)) - Removed a lock true On Tue, Jun 7, 2016 at 9:01 PM, Alan Gates m

Re: Hive Hcatalog Streaming. Why hive table must be bucketed?

HCatalog streaming works with Hive's transactional tables which are currently required to be bucketed. The later is to improve read performance since these tables also support update/delete operations (though not via streaming). "The number of buckets is ideally ..." this is obsolete (as of HIVE

Re: Error selecting from a Hive ORC table in Spark-sql

The system thinks t2 is an Acid table but the files on disk don’t follow the convention acid system would expect. Perhaps Xuefu Zhang would know more on Spark/Aicd integration. From: Mich Talebzadeh mailto:mich.talebza...@gmail.com>> Reply-To: "user@hive.apache.org"

Re: [ANNOUNCE] New Hive Committer - Wei Zheng

Congratulations! From: Hari Sivarama Subramaniyan mailto:hsubramani...@hortonworks.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Thursday, March 10, 2016 at 10:02 AM To: "user@hive.apache.org" mailto:user@hiv

Re: Synchronizing Hive metastores across clusters

Metastore supports MetaStoreEventListener and MetaStorePreEventListener which may be useful here Eugene From: Elliot West mailto:tea...@gmail.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Thursday, December 17, 2015 at 8:21 AM To: "user@

Re: Connection between TempletonJob and Worker Nodes remains in FIN_WAIT_2 state for long time

https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+Hive lets you submit a file with SQL statements Eugene On 12/11/15, 4:09 PM, "mahender bigdata" wrote: >Hi, >We have submitted too many jobs to webhcat (templeton) reason is our HQL >has multiple hive statements,each hive state

Re: Adding a new column to a table and updating it

stood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.

Re: Adding a new column to a table and updating it

Schema evolution is not supported for Acid tables. https://issues.apache.org/jira/browse/HIVE-11981 fixes it but hasn't been released yet. There is no quick way to recover data. You could write a script to use ORC FileDump utility to look at actual files in the table to group them into sets where

Re: Hive 1.2.1 and concurrency

This doesn't directly answer the question but may be useful. The tables in hive-txn.schema-* are only relevant if hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager and hive.suport.concurrency=true. If you only set hive.suport.concurrency=true, you'll be using the lock manager set

Re: Need you Expert inputs

instead of using " SET hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager" can add these props to your hive-site.xml and try again? Eugene From: Mahender SNPC mailto:mails2mahen...@hotmail.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apa

Re: [ANNOUNCE] New PMC Member : John Pullokkaran

Congrats! From: Sergey Shelukhin mailto:ser...@hortonworks.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Tuesday, November 24, 2015 at 3:13 PM To: "user@hive.apache.org" mailto:user@hive.apache.org>>, "d...@h

Re: hive transaction strange behaviour

pactions running on other tables/partitions, things must be configured properly. If you use Alter Table to launch compaction, it should just do it, w/o paying attention to thresholds. From: Eugene Koifman mailto:ekoif...@hortonworks.com>> Reply-To: "user@hive.apache.org&

Re: hive transaction strange behaviour

he partition where delta files still exists.why compactor has not picked the other partition, when and how these partition will be picked up for compaction. Thanks On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: When Compaction process runs, it will

Re: hive transaction strange behaviour

When Compaction process runs, it will create base directory. https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration at a minimum you need hive.compactor.initiator.on

Re: DML operations and transactions in Hive

https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions describes this in detail Assuming all tables in the query are ACID tables, then reads are consistent. ACID is supported by MVCC architecture at the storage layer (ORC). Locking is at partition level. If one statement is trying t

Re: Locking when using the Metastore/HCatalog APIs.

Wouldn't it make more sense for the api to acquire required locks automatically? That seems like a simpler user model. From: Alan Gates mailto:alanfga...@gmail.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Tuesday, October 27, 2015 at 1

Re: locks are held on tables even when no job running

Mich, how were you running/killing the job? was it ^C of CLI or something else? (The only time you’d get Exclusive lock is to drop an object. (With DbTxnManager which looks like what you are using)) The locks will timeout but https://issues.apache.org/jira/browse/HIVE-11317 may be relevant. Fu

Re: Transaction deadlocks

could you file a Jira for this please? From: Eugene Koifman mailto:ekoif...@hortonworks.com>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>> Date: Wednesday, September 23, 2015 at 8:15 AM To: "user@hive.apach

Re: Transaction deadlocks

rruptedException ex) {} throw new RetryException(); } LOG.error("Fatal error. Retry limit (" + this.retryLimit + ") reached. Last error: " + getMessage(e)); this.retryNum = 0; } else { this.deadlockCnt = 0; this.retryNum = 0;

Re: Transaction deadlocks

What version of Hive are you running? In the current codebase at least, TxnHandler.checkRetryable() keeps track of how many retries it allowed – it will not retry forever. From: Steve Howard mailto:stevedhow...@gmail.com>> Reply-To: "user@hive.apache.org" mailto:u

Re: UPDATE and DELETE fail due to Error 10122 on Hive 1.2.1

use 'transactional', I.e. lower case From: , Jian-bing mailto:jian-bin@sap.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Monday, September 7, 2015 at 11:58 PM To: "user@hive.apache.org" mailto:user@hive.ap

Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan

Congrats! From: Pengcheng Xiong mailto:pxi...@apache.org>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Wednesday, September 16, 2015 at 1:23 PM To: "user@hive.apache.org" mailto:user@hive.apache.org>> Cc: "d...@hi

Re: Error communicating with metastore

could you execute "set hive.txn.manager" and "set hive.metastore.uris" in Hive CLI to confirm that your client is configured correctly? From: Sarath Chandra mailto:sarathchandra.jos...@algofusiontech.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org

Re: hive -e run tez query error

perhaps deleteOnExit() is set somewhere From: Jeff Zhang mailto:zjf...@gmail.com>> Reply-To: User mailto:user@hive.apache.org>> Date: Thursday, June 25, 2015 at 2:09 AM To: User mailto:user@hive.apache.org>> Subject: Re: hive -e run tez query error The drop table command between these 2 query fai

Re: delta file compact take no effect

Delta files that are no longer needed are deleted asynchronously. For example, you may have some query using delta_002_002. A minor compaction, for example, can run concurrently and create delta_001_003 but it will leave delta_001_001, delta_002_002, delta_003_

Re: hive sql on tez run forever

This isn’t a valid rewrite. if a(x,y) has 1 row (1,2) and b(x,z) has 1 row (1,1) then the 1st query will produce 1 row but the 2nd query with subselects will not. On 5/11/15, 10:13 AM, "Gopal Vijayaraghavan" wrote: >Hi, > >> I change the sql where condition to (where t.update_time >= >>'2015-05-

Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Can you provide precise table definitions (DDL, describe formatted, etc and statements you are running (with explain plan)? (the simplest version that reproduces the issue) Also your hive-site.xml. Would be even better if you file a JIRA with the same. You are clearly hitting a bug but it's im

Re: FAILED: LockException [Error 10280]: Error communicating with the metastore

looking at the code, TxnHandler.checkQFileTestHack should only do anything if hive.in.test or hive.in.tez.test is true which I think is only used by unit tests From: "Grant Overby (groverby)" mailto:grove...@cisco.com>> Reply-To: "user@hive.apache.org" mailto:user@h

Re: ACID ORC file reader issue with uncompacted data

rg>> Subject: Re: ACID ORC file reader issue with uncompacted data Thanks for the heads-up and use case validation. In the case of the file names, what function does the additional id perform (presuming the first two are still transaction id bounds)? On 29 April 2015 at 18:37, Eugene Koifman

Re: ACID ORC file reader issue with uncompacted data

This is not an answer to your question, but FYI. The work in https://issues.apache.org/jira/browse/HIVE-9675 will change how the delta files are named which may affect your work. Once that work is complete, the deltas will be named delta_xxx_yyy_zz, so you may have delta_002_002_1,delta_002_002

Re: create statement is not working

:user@hive.apache.org>> Subject: Re: create statement is not working in my case it just appeared as a hang. I understood the problem now i want to know how i can avoid this. On Sat, Apr 25, 2015 at 3:24 AM, Eugene Koifman mailto:ekoif...@hortonworks.com>> wrote: No, I was suggesting

Re: create statement is not working

ubsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Eugene Koifman [mailto:ekoif...@hortonworks.com] Sent: 24

Re: create statement is not working

Do you have hive.support.concurrency=true and long running insert overwrite statements running concurrently? If so, you may be hitting something something like https://issues.apache.org/jira/browse/HIVE-10242 From: Mich Talebzadeh mailto:m...@peridale.co.uk>> Reply-To: "user@hive.apache.org

Re: UDF cannot be found when the query is submitted via templeton

can you give the complete REST call you are making to submit the query? From: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Sunday, April 19, 2015 at 8:23 PM To: "user@hive.apache.org

Re: [ANNOUNCE] New Hive Committer - Mithun Radhakrishnan

Congrats! From: Vaibhav Gumashta mailto:vgumas...@hortonworks.com>> Reply-To: "user@hive.apache.org" mailto:user@hive.apache.org>> Date: Wednesday, April 15, 2015 at 5:08 PM To: "user@hive.apache.org" mailto:user@hive.apache.org>>, venka

Re: Create tables is blocked while there are queries executing in a specific database in Hive

Do you have hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager in addition to hive.support.concurrency=true? if so you may be hitting https://issues.apache.org/jira/browse/HIVE-10242. From: Guillermo Ortiz mailto:konstt2...@gmail.com>> Reply-To: "user@hive.apache.org

Re: Concurrency issue, Setting hive.txn.manager to "org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"

can you check that the schema in your metastore db has transaction related tables? You can find the list of tables in hive-txn-schema-0.14.0.mysql.sql, for example. From: Mich Talebzadeh mailto:m...@peridale.co.uk>> Reply-To: "user@hive.apache.org" mailto:user@hive.

Re: Can WebHCat show non-MapReduce jobs?

https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+Jobs should produce all jobs (assuming the calling user has permissions to see them). templeton.Server.showJobList() has detailed JavaDoc From: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Reply-To: "user@hive.apache.org

Re: Submitting via WebHCat won't put hive log into ATS?

I'm working https://issues.apache.org/jira/browse/HIVE-10066 where I plan to make all of hive-site.xml be available to webhcat's LaunchMapper so that these settings don't have to be duplicated in templeton.hive.properties. From: Lefty Leverenz mailto:leftylever...@gmail.com>> Reply-To: "user@hiv

Re: [ANNOUNCE] New Hive PMC Member - Prasad Mujumdar

Congratulations! On Wed, Dec 10, 2014 at 9:17 AM, Vikram Dixit wrote: > Congrats Prasad! > On Dec 10, 2014 8:21 AM, "Naveen Gangam" wrote: > >> Congratulations Prasad. Way to go. >> >> On Wed, Dec 10, 2014 at 9:39 AM, Dapeng, Sun >> wrote: >> >>> Congrats! >>> >>> >>> >>> Regards >>> >>> Dape

Re: [ANNOUNCE] New Hive PMC Member - Alan Gates

Congratulations! On Mon, Oct 27, 2014 at 3:51 PM, Xiaobing Zhou wrote: > Alan, congrats! > > On Mon, Oct 27, 2014 at 3:45 PM, Xuefu Zhang wrote: > > > Congratulations, Alan! > > > > On Mon, Oct 27, 2014 at 3:43 PM, Matthew McCline < > mmccl...@hortonworks.com > > > > > wrote: > > > > > Congratu

Re: [ANNOUNCE] New Hive Committer - Eugene Koifman

, at 3:46 PM, Xiaobing Zhou >>> wrote: >>> >>> > Congrats, Eugene! >>> > >>> > On Fri, Sep 12, 2014 at 3:35 PM, Carl Steinbach >>> wrote: >>> > >>> >> The Apache Hive PMC has voted to make Eugene Koifman a committer on

Re: WebHCat and MapReduce JAR

g.class > > > > Thanks, > > > > *Remy Dubois* > *Component team manager and Big Data principal * > *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email * > rdub...@talend.com | *Skype *remy.dubois.talend > *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France |

Re: WebHCat and MapReduce JAR

libjars as form param should work. Is there anything relevant in webhcat.log? What version of webhcat are you using? What is the exact REST call you are making? On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois wrote: > Hi, > > > > I’m trying to submit a MapReduce job through webhcat using the >

Re: hive auto join conversion

would manually rewriting the query from (T1 union all T2) LOJ S to equivalent (T1 LOJ S) union all (T2 LOJ S) help work around this issue? On Wed, Jul 30, 2014 at 6:19 PM, Chen Song wrote: > I tried that and I got the following error. > > FAILED: SemanticException [Error 10227]: Not all clauses

Re: java.lang.TypeNotPresentException: Type timestamp not present

instead of org.apache.hcatalog.pig.HCatStorer() use org.apache.hive.hcatalog.pig.HCatStorer() On Mon, Jun 2, 2014 at 12:54 PM, Rahul Channe wrote: > Try loading the data in a temporary hive table with datatype as String, > then you can populate the data in the final table > > insert into table

Re: Writing to an ORCFile using MapReduce + HCatalog APIs

If you are writing from Pig using HCatStorer you don't need to create HCatSchema. https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore#HCatalogLoadStore-Usage.1has examples on how to do it. So if you create a Hive table that use ORC you should be able to write your Pig cursor to tha

Re: org.apache.hadoop.hive.metastore.HiveMetaStoreClient with webhcat REST

the URL to describe a table should be .../database//table/? but your exception happens before the URL problem. Have you checked templeton.hive.properties property in webhat-site.xml? does hive.metastore.uris there point at the right location for your metastore? templeton.libjars is translate

Re: [ANNOUNCE] New Hive Committers - Sergey Shelukhin and Jason Dere

Congratulations Sergey and Jason! On Mon, Jan 27, 2014 at 10:58 AM, Thejas Nair wrote: > Congrats Jason and Sergey! > Well deserved! > Looking forward to your help in getting the patch available counts > down (its at 225 now)! > > > On Mon, Jan 27, 2014 at 10:55 AM, Vaibhav Gumashta > wrote: >

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

Congratulations! On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner < ghagleit...@hortonworks.com> wrote: > Congratulations Vikram! > > Thanks, > Gunther. > > > On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan < > hsubramani...@hortonworks.com> wrote: > >> congrats Vikram!! >> >> >> >> >> On Mo

Re: WebHCat MapReduce Job Syntax

lExecService' output in the jobtracker or tasktracker > logs. We are using hive 0.11 though so maybe not set to DEBUG? > > > On Mon, Dec 30, 2013 at 2:11 PM, Eugene Koifman > wrote: > >> Is there any output from TrivialExecService class in any hadoop logs? >>

Re: WebHCat MapReduce Job Syntax

20:33:43,157 | org.apache.hcatalog.templeton.Server | > queued job job_201312212124_0166 in 300 ms > > I still the same behavior with just the TempletonControllerJob getting > kicked off and ending successfully without the Camus job starting. I > didn't see any errors in the jobtrack

Re: WebHCat MapReduce Job Syntax

have you tried adding -d arg=-P before -d arg=/tmp/properites On Mon, Dec 30, 2013 at 11:14 AM, Jonathan Hodges wrote: > Sorry accidentally hit send before adding the lines from webhcat.log > > DEBUG | 30 Dec 2013 19:08:01,042 | org.apache.hcatalog.templeton.Server | > queued job job_20131

Re: Using Hive with WebHCat

1_0020/callback >> >> Those files indeed don't exist. The job_201311281741_0020 directory was >> created after starting the Hive job, but only has 'completed', 'exitValue', >> and 'user' files. Could these missing files be the issue? If so, h

Re: Using Hive with WebHCat

It may be worth looking in webhcat.log and using job tracker UI On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges wrote: > Hi, > > I have setup WebHCat that is bundled with Hive 0.11.0. I am able to kick > of map reduce jobs with the REST API successfully. However I am having > some issues with

Re: hcatalog takes minutes talking to mysql metadata

perhaps HIVE-4914 relevant On Wed, Aug 28, 2013 at 3:11 AM, Michał Czerwiński wrote: > Also what is worth mentioning I have tried running 0.4.0-cdh4.3.0-SNAPSHOT > jars (from > https://repository.cloudera.com/content/groups/public/org/apache/hcatalog/hcatalog-core/) > with exactly the same issu

Re: [ANNOUNCE] New Hive Committer - Thejas Nair

Congrats Thejas! On Tue, Aug 20, 2013 at 3:31 AM, Carl Steinbach wrote: > The Apache Hive PMC has voted to make Thejas Nair a committer on the Apache > Hive project. > > Please join me in congratulating Thejas! > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the in