RE: LLAP - Hive on Tez - ERROR on SQL query

on Tez - ERROR on SQL query Hi. I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works. When I run a SQL query like select count(*) from database group by column; I've got the following error: Caused by: java.lang.IllegalStateException

LLAP - Hive on Tez - ERROR on SQL query

Hi. I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works. When I run a SQL query like select count(*) from database group by column; I've got the following error: Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkSt

Hive on Tez : yarn logs missing timestamp

Hi, We are using Hive on Tez (see versions below) and aren't able to get TezChild class to log the timestamp even though tez-container-log4j.properties has the ISO time in the logger pattern. Sample Logs: [TezChild] INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator - FS[3]: records wr

Hive on Tez vs LLAP Count difference

Hi. So we encountered this issue where counts on transactional tables are sometimes different between Hive on Tez vs LLAP. We're using Hive 3.1.0 BTW, maybe there's already a fix for this or a workaround. Thanks in advance. -Bernard

Re: Hive on Tez vs Impala

> I wish the Hive team to keep things more backward-compatible as well. Hive is > such an enormous system with a wide-spread impact so any > backward-incompatible change could cause an uproar in the community. The incompatibilities were not avoidable in a set of situations - a lot of those we

Re: Hive on Tez vs Impala

I'm using Hive 3.1 on Tez/LLAP and I must say the experience was not good but it was worth it. We built Hive from HDP's hive-release and add Tez UI back, combined that with Hue 4.3 (also built from Cloudera Hue). Now that the two companies have merged I think things are going to get better (I'm not

Re: Hive on Tez vs Impala

I have changes jobs 3 times since tez was introduced. It is a true waste of compute resources and time that it was never patched in. So I either have to waste my time patching it in, waste my time running a side deployment, or not installing it and waste money having queries run longer on mr/spark

Re: Hive on Tez vs Impala

If we install our own build of Hive, we'll be out of support from CDH. Tez is not supported anyway and we're not touching any CDH bits, so it's not a big issue to have our own build of Tez engine. > On Apr 15, 2019, at 9:20 PM, Gopal Vijayaraghavan wrote: > > > Hi, > >>> However, we have bu

Re: Hive on Tez vs Impala

Hi, >> However, we have built Tez on CDH and it runs just fine. Down that path you'll also need to deploy a slightly newer version of Hive as well, because Hive 1.1 is a bit ancient & has known bugs with the tez planner code. You effectively end up building the hortonworks/hive-release build

Re: Hive on Tez vs Impala

e: >> >>> I tested the performance of Impala 2.12.0+cdh5.15.2+0 in Cloudera CDH >>> 5.15.2 a while ago. I compared it with Hive 3.1.1 on MR3 (where MR3 is a >>> new execution engine for Hadoop and Kubernetes). You can find the result at: >>> >&

Re: Hive on Tez vs Impala

5.2 a while ago. I compared it with Hive 3.1.1 on MR3 (where MR3 is a >> new execution engine for Hadoop and Kubernetes). You can find the result at: >> >> https://mr3.postech.ac.kr/blog/2019/03/22/performance-evaluation-0.6/ >> >> On average, Hive on MR3 is about 30% fa

Re: Hive on Tez vs Impala

. You can find the result at: > > https://mr3.postech.ac.kr/blog/2019/03/22/performance-evaluation-0.6/ > > On average, Hive on MR3 is about 30% faster than Hive on Tez on sequential > queries. For concurrent queries, the throughput of Hive on MR3 is about > three times higher th

Re: Hive on Tez vs Impala

le ago. I compared it with Hive 3.1.1 on MR3 (where MR3 is a > new execution engine for Hadoop and Kubernetes). You can find the result at: > > https://mr3.postech.ac.kr/blog/2019/03/22/performance-evaluation-0.6/ > > On average, Hive on MR3 is about 30% faster than Hive on Tez on sequ

Re: Hive on Tez vs Impala

average, Hive on MR3 is about 30% faster than Hive on Tez on sequential queries. For concurrent queries, the throughput of Hive on MR3 is about three times higher than Hive on Tez (when tested with 16 concurrent queries). You can find the result at: https://mr3.postech.ac.kr/blog/2018/10/30

Hive on Tez vs Impala

Hi, We are using CDH 5, with Impala 2.7.0-cdh5.9.1 and Hive 1.1 (MapReduce) I can't find the info regarding Hive on Tez performance compared to Impala. Does someone know or compared it? Thanks Artur Sukhenko

Re: Announce: MR3 0.3, and performance comparison with Hive-LLAP, Presto, Spark, Hive on Tez

Thank you for reading the article. I plan to publish the result of running concurrent queries with the release of MR3 0.4 (which implements a feature affecting the performance of concurrent queries). --- Sungwoo On Sat, Sep 8, 2018 at 8:06 AM Nicolas Paris wrote: > > On Thu, Aug 16, 2018 at 10:

Re: Announce: MR3 0.3, and performance comparison with Hive-LLAP, Presto, Spark, Hive on Tez

On Thu, Aug 16, 2018 at 10:55:19PM +0900, Sungwoo Park wrote: > The article compare the following six systems: Great article, as usual. Would have been great to also compare concurrent queries. In particular, I guess presto on that point perform the best. That metric is major since such technol

Re: Announce: MR3 0.3, and performance comparison with Hive-LLAP, Presto, Spark, Hive on Tez

The article can be found at: https://mr3.postech.ac.kr/blog/2018/08/15/comparison-llap-presto-spark-mr3/ -- Sungwoo Park On Thu, Aug 16, 2018 at 10:53 PM, Sungwoo Park wrote: > Hello Hive users, > > I am pleased to announce the release of MR3 0.3. A new feature of MR3 0.3 > is its support for

Announce: MR3 0.3, and performance comparison with Hive-LLAP, Presto, Spark, Hive on Tez

Hello Hive users, I am pleased to announce the release of MR3 0.3. A new feature of MR3 0.3 is its support for Hive 3.0.0 on Hadoop 2.7/2.8/2.9. I have also published a blog article that uses the TPC-DS benchmark to compare the following six systems: 1) Hive-LLAP included in HDP 2.6.4 2) Presto 0

Running commands on Hive on Tez failed with NPE

Hi Hive community, I recently encountered with a problem while running command in Hive on Tez as execution engine. The command failed with below stack trace Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException

Re: Hive on Tez local debug

bash> hive --hiveconf hive.root.logger=DEBUG,console On Thu, Jan 18, 2018 at 9:16 PM, Jia, Ke A wrote: > Hi all, > > How debug execution code in hive on tez? “hive --debug” command only debug > the explain level code and does not debug the execution code. Does hot have > th

Hive on Tez local debug

Hi all, How debug execution code in hive on tez? "hive --debug" command only debug the explain level code and does not debug the execution code. Does hot have the similar usage as hos by using command "set spark.master=local;"? Thanks for your help. Regards, Jia Ke

Re: How to measure the execution time of query on Hive on Tez

590,200 22,331 107,740,470 76 > 2017-10-12T16:29:39,318 INFO [main] SessionState: Reducer 9 > 8070.00 24,630249 764 > 2017-10-12T16:29:39,318 INFO [main] SessionState: > --- > > From: Zh

RE: How to measure the execution time of query on Hive on Tez

49 764 2017-10-12T16:29:39,318 INFO [main] SessionState: --- From: Zhang, Liyun [mailto:liyun.zh...@intel.com] Sent: Thursday, October 12, 2017 4:40 PM To: d...@hive.apache.org Subject: How to measure th

Re: Hive on Tez: Tez taking nX more containers than Mapreduce for union all

Hi Gopal, Please find the answers inline. On Fri, Mar 17, 2017 at 9:01 PM, Gopal Vijayaraghavan wrote: > > > We are using a query with union all and groupby and same table is read > multiple times in the union all subquery. > … > > When run with Mapreduce, the job is run in one stage consuming

Re: Hive on Tez: Tez taking nX more containers than Mapreduce for union all

> We are using a query with union all and groupby and same table is read > multiple times in the union all subquery. … > When run with Mapreduce, the job is run in one stage consuming n mappers and > m reducers and all union all scans are done with the same job. The logical plans are identical

Hive on Tez: Tez taking nX more containers than Mapreduce for union all

Hi, We are migrating our hive queries from Mapreduce to Tez . We are using a query with union all and groupby and same table is read multiple times in the union all subquery. We have noticed a issue with tez here, it runs with kX times more tasks than MR where k is the number of union alls in the

Re: [Hive on Tez] Running queries in tez non-session mode not working

Hi Gopal, I have tried setting by both the means, setting after launching cli via set command and setting before launching cli using hive --hiveconf tez.am.mode.session=false. Both didn't work. Thanks, Ravi On Wed, Mar 15, 2017 at 12:44 AM, Gopal Vijayaraghavan wrote: > > > > by setting tez.am

Re: [Hive on Tez] Running queries in tez non-session mode not working

> by setting tez.am.mode.session=false in hive-cli and hive-jdbc via > hive-server2. That setting does not work if you do "set tez.am.*" parameters (any tez.am params). Can you try doing hive --hiveconf tez.am.mode.session=false instead of a set; param and see if that works? Cheers, Gopa

[Hive on Tez] Running queries in tez non-session mode not working

Hi, We wanted to run our hive queries in non-session mode as we have a problem tracking the app ids. Until our tracking is fixed we wanted to run each dag in a separate app master. I have read that running tez in non-session mode will achieve this functionality, by setting *tez.am.mode.session*=f

Re: Some Hive on Tez queries don't finish

Are there are any exceptions seen in the app logs (you can ignore the Interrupted exceptions in the logs as you killed the job). It would be helpful if you can share the app logs. ~Rajesh.B On Mon, Nov 28, 2016 at 2:53 PM, Premal Shah wrote: > Hi, > We've been running Hive 2.0.1 on Tez 0.8.4 fo

Some Hive on Tez queries don't finish

Hi, We've been running Hive 2.0.1 on Tez 0.8.4 for a few weeks now. Most queries that we run work. However some queries that go over millions to billions of rows don't finish using Tez as the execution engine. Here's an example of a simple query that does not finish select count(distinct external

Re: Hive on Tez CTAS query breaks

Sergey and Gopal, Tried it will the alias. Still breaks. CREATE TABLE unique_ip_tmp AS SELECT DISTINCT new.ip FROM ( SELECT COUNT(0) as count , ip as ip FROM t1 WHERE dt BETWEEN '2016-11-08' AND '2016-11-08' GROUP BY ip UNION

Re: Hive on Tez CTAS query breaks

> Thanx for the suggestion. It works with the setting you suggested. > > What does this mean? Do I need to special case this query. You need to report a bug on https://issues.apache.org/jira/browse/HIVE Because, this needs to get fixed. > Turning off CBO cluster-wide won't be the right thing to

Re: Hive on Tez CTAS query breaks

"user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>> Date: Wednesday, November 9, 2016 at 23:16 To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>> Subject: Re: Hive on Tez CTAS query breaks

Re: Hive on Tez CTAS query breaks

Hi Gopal, Thanx for the suggestion. It works with the setting you suggested. What does this mean? Do I need to special case this query. Also, I am trying different things to see what is breaking. Looks like I have a UNION ALL and both sides have a query with a GROUP BY. This breaks. CREATE TABL

Re: Hive on Tez CTAS query breaks

> If I run a query with CREATE TABLE AS, it breaks with the error below. > However, just running the query works if I don't try to create a table from > the results. It does not happen to all CTAS queries.  Not sure if that's related to Tez at all. Can try running it with set hive.cbo.enable

Hive on Tez CTAS query breaks

Hi, If I run a query with CREATE TABLE AS, it breaks with the error below. However, just running the query works if I don't try to create a table from the results. It does not happen to all CTAS queries. 2016-11-10T04:52:27,531 ERROR [cb5f35ff-f404-41ce-aa9b-87708d80b078 main]: ql.Driver (Session

Re: Hive on TEZ + LLAP

Sounds like if I am correct joining a fact table store_sales; with two dimensions? cool thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Hive on TEZ + LLAP

> What was the type (Parquet, text, ORC etc) and row count for each three >tables above? I always use ORC for flat columnar data. ORC is designed to be ideal if you have measure/dimensions normalized into tables - most SQL workloads don't start with an indefinite depth tree. hive> select count(1

Re: Hive on TEZ + LLAP

Thanks In this sample query select i_brand_id brand_id, i_brand brand, sum(ss_ext_sales_price) ext_price from *date_dim, store_sales, item * where date_dim.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and i_manager_id=36 and

Re: Hive on TEZ + LLAP

> These looks pretty impressive. What execution mode were you running >these? Yarn client may be? There is no other mode - everything runs on YARN. > 53 times The factor is actually bigger in actual execution. The MRv2 version takes 2.47s to prep a query, while the LLAP version takes 1.64s.

Re: Hive on TEZ + LLAP

Also has there been simple benchmarks to compare: > > > > 1. Hive on MR > > 2. Hine on Tez > > 3. Hive on Tez with LLAP > > I ran one today, with a small BI query in my test suite against a 1Tb > data-set. > > TL;DR - MRv2 (203.317 seconds), Tez (13.681s), LLAP

Re: Hive on TEZ + LLAP

> Also has there been simple benchmarks to compare: > > 1. Hive on MR > 2. Hine on Tez > 3. Hive on Tez with LLAP I ran one today, with a small BI query in my test suite against a 1Tb data-set. TL;DR - MRv2 (203.317 seconds), Tez (13.681s), LLAP (3.809s). *Warning*: This is n

Re: Hive on TEZ + LLAP

Hi, This is interesting. Are there any late presentations of Hive on Tez and Hive on Tez with LLAP. Also has there been simple benchmarks to compare: 1. Hive on MR 2. Hine on Tez 3. Hive on Tez with LLAP It would be interesting how these three fare. Thanks Dr Mich Talebzadeh

Re: Hive on TEZ + LLAP

> I have also heard about Hortonworks with Tez + LLAP but that is a distro? Yes. AFAIK, during Hadoop Summit there was a HDP 2.5 techpreview sandbox instance which shipped Hive2 (scroll down all the way to end in the downloads page). Enable the "interactive mode" in Ambari for a HiveServer2 conf

Re: Hive on TEZ + LLAP

HDP 2.5 includes LLAP. Cheers, Andrew On Fri, Jul 15, 2016 at 11:36 AM, Jörn Franke < jornfra...@gmail.com [jornfra...@gmail.com] > wrote: I would recommend a distribution such as Hortonworks were everything is already configured. As far as I know llap is currently not part of any distribution.

Re: Hive on TEZ + LLAP

7626288998 div.yiv7626288998WordSection1 {}#yiv7626288998 Amazon AWS has recently released EMR with Hive + Tez as well.   Cheers Andrew   From: Jörn Franke Reply-To: "user@hive.apache.org" Date: Friday, July 15, 2016 at 8:36 AM To: "user@hive.apache.org" Subject: Re:

Re: Hive on TEZ + LLAP

Amazon AWS has recently released EMR with Hive + Tez as well. Cheers Andrew From: Jörn Franke Reply-To: "user@hive.apache.org" Date: Friday, July 15, 2016 at 8:36 AM To: "user@hive.apache.org" Subject: Re: Hive on TEZ + LLAP I would recommend a distribution such

Re: Hive on TEZ + LLAP

I would recommend a distribution such as Hortonworks were everything is already configured. As far as I know llap is currently not part of any distribution. > On 15 Jul 2016, at 17:04, Ashok Kumar wrote: > > Hi, > > Has anyone managed to make Hive work with Tez + LLAP as the query engine in >

Hive on TEZ + LLAP

Hi, Has anyone managed to make Hive work with Tez + LLAP as the query engine in place of Map-reduce please? If you configured it yourself which version of Tez and LLAP work with Hive 2. Do I need to build Tez from source for example Thanks

Re: Anyone successfully deployed Hive on TEZ engine?

Hi Gopal, please see my correspondence about Tez in tez user group. I forwarded to hive user group. thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Anyone successfully deployed Hive on TEZ engine?

> I do not use any vendor's product., All my own set up, build and >configure. My autobuild scripts should serve as readable documentation for this, since nearly everything's in a single Makefile with an install: target. Or take the easy route with $ make dist install In case you use the llap b

Re: Anyone successfully deployed Hive on TEZ engine?

Thanks Gopal. I do not use any vendor's product., All my own set up, build and configure. No CDH no HDL etc. This the current stack that I have: Java *java -version* java version "1.8.0_77" Java(TM) SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03,

Re: Anyone successfully deployed Hive on TEZ engine?

> In short at the simplest set up what Resource Manager it works with? Tez+Hive needs HDFS and YARN 2.6.0+ (preferably as close to an Apache build as possible - CDH clusters need more work). Hive2 needs Apache Slider 0.91 right now, to start the cache daemons on YARN (see SLIDER-82). > If so ki

Re: Anyone successfully deployed Hive on TEZ engine?

thanks Damien. I tried TEZ 0.82 with Hive 2 although I did not persevere. When you say "Not stable" are you referring to using it with YARN etc. In short at the simplest set up what Resource Manager it works with? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?i

Re: Anyone successfully deployed Hive on TEZ engine?

HIVE 1.2.1 and Tez 0.5.2 or 0.7.0 works pretty well beginning to use HIVE 2.0.0 and 0.8.x but not stable :/ 2016-05-29 22:26 GMT+02:00 Mich Talebzadeh : > > Please bear in mind that I am talking about your own build not anything > comes as part of Vendor's package. > > If so kindly specify both

Anyone successfully deployed Hive on TEZ engine?

Please bear in mind that I am talking about your own build not anything comes as part of Vendor's package. If so kindly specify both Hive and TEZ versions. Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: hive on tez hadoop-common version problem.

> I have some problem with hive-on-tez. > email thread below is forwarding originally wrote to tez users. AFAIK, this problem only happens with CDH and never with pure Apache bigtop builds. Neither minimal JAR nor the cluster libs work as the problem is with the cluster jar ABIs

hive on tez hadoop-common version problem.

Hi all. I have some problem with hive-on-tez. email thread below is forwarding originally wrote to tez users. Could someone please check this problem out? Any advice will be appreciated. thanks. -- Forwarded message -- From: no jihun Date: 2016-02-15 17:04 GMT+09:00 Subject

Re: NPE when reading Parquet using Hive on Tez

> I dug a little deeper and it appears that the configuration property >"columns.types", which is used >org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(), > is not being set. When I manually set that property in hive, your >example works fine. Good to know more about the NPE

Re: NPE when reading Parquet using Hive on Tez

on my current build with simple types and it looks like >> the issue needs more detail on the column types for a repro. >> >> hive> create temporary table x (x int) stored as parquet; >> hive> insert into x values(1),(2); >> hive> select count(*) from x w

hive on tez serialization and deserialization( custom Serde initialize() just called one time in hive client, when split in AM)

Hello,all:       【My environment versions are ：Hadoop 2.6.0 、hive 1.2.1、tez 0.7.0】 Our term develop a plug-in in hive, its function is similiar to hive-hbase-handler. Now I executed a HQL “select count(*) from h_im;”(h_im is an external table, hbase table) in hive CLI, it thro

RE: Hive on TEZ fails starting

re that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: 06 January 2016 15:03 To: user@hive.apache.org Subject: RE: Hive on TEZ fails starting Not starting at all!

RE: Hive on TEZ fails starting

It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Rajesh Balamohan [mailto:rajesh.balamo...@gmail.com] Sent: 06 January 2016 09:18 To: user@hive.apache.or

Re: Hive on TEZ fails starting

dorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Ltd, its subsidiaries nor their employees > accept any responsibili

RE: Hive on TEZ fails starting

his email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Rajesh Balamohan [mailto:rajesh.balamo...@gmail.com <mailto:rajesh.balamo...@gmail.com> ] Sent: 05 January 2016 00:35 To: user@hive.apache.org <mailto:user@hive.apache.

RE: Hive on TEZ fails starting

he designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the

RE: Hive on TEZ fails starting

ponsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Rajesh Balamohan [mailto:rajesh.balamo...@gmail.com] Sent: 05 January 2016 11:46 To: user@hive.apache.org Subject: R

Re: NPE when reading Parquet using Hive on Tez

e > the issue needs more detail on the column types for a repro. > > hive> create temporary table x (x int) stored as parquet; > hive> insert into x values(1),(2); > hive> select count(*) from x where x.x > 1; > Status: DAG finished successfully in 0.18 second

Re: Hive on TEZ fails starting

ion in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > L

RE: Hive on TEZ fails starting

the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Rajesh Balamohan [mailto:rajesh.balamo...@gmail.com] Sent: 05 January 2016 00:35 To: user@hive.apache.org Subjec

Re: Hive on TEZ fails starting

estroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free,

RE: Hive on TEZ fails starting

ted. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Rajesh Balamohan [mailto:rajesh.balamo...@gmail.com] Sent: 05 January 2016 00:07 To: user@hive.apache.org Su

Re: Hive on TEZ fails starting

Can you try removing double-quotes for "tez.lib.uris" in tez-site.xml (i.e just use hdfs://rhes564:9000/apps/tez-0.7.1-SNAPSHOT/tez-0.7.1- SNAPSHOT.tar.gz)? ~Rajesh.B On Tue, Jan 5, 2016 at 5:30 AM, Mich Talebzadeh wrote: > Hi, > > > > Trying to run Hive on TEZ fo

Hive on TEZ fails starting

Hi, Trying to run Hive on TEZ for the first time. Getting the error below 0: jdbc:hive2://rhes564:10010/default> set hive.execution.engine=tez; No rows affected (0.001 seconds) 0: jdbc:hive2://rhes564:10010/default> use oraclehadoop; No rows affected (0.018 secon

Re: NPE when reading Parquet using Hive on Tez

x.x > 1; Status: DAG finished successfully in 0.18 seconds OK 1 Time taken: 0.792 seconds, Fetched: 1 row(s) hive> Do you have INT96 in the schema? > I'm currently evaluating Hive on Tez as an alternative to keeping the >SparkSQL thrift sever running all the time locking up resource

NPE when reading Parquet using Hive on Tez

Hi, When I perform any operation on a data set stored in Parquet format using Hive on Tez, I get an NPE (see bottom for stack trace). The same operation works fine on tables stored as text, Avro, ORC and Sequence files. The same query on the parquet tables also works fine if I use Hive on MR

Re: Is YSmart integrated into Hive on tez ?

gt; >>> Reducer 3 <- Reducer 2 (SIMPLE_EDGE) > > On Tue, Sep 1, 2015 at 1:14 AM, Pengcheng Xiong wrote: > >> Hi Jeff, >> >> From code base point of view, YSmart is integrated into Hive on Tez >> because it is one of the optimization of the curren

Re: Is YSmart integrated into Hive on tez ?

), Map 4 (SIMPLE_EDGE) >>> Reducer 3 <- Reducer 2 (SIMPLE_EDGE) On Tue, Sep 1, 2015 at 1:14 AM, Pengcheng Xiong wrote: > Hi Jeff, > > From code base point of view, YSmart is integrated into Hive on Tez > because it is one of the optimization of the current Hive. However,

Re: Is YSmart integrated into Hive on tez ?

Hi Jeff, From code base point of view, YSmart is integrated into Hive on Tez because it is one of the optimization of the current Hive. However, from the execution point of view, it is now disabled when Hive is running on Tez. You may take look at the source code of Hive Optimizer.java

Re: Hive on tez error

che.org>> Date: Friday, August 28, 2015 at 12:33 AM To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>>, "u...@tez.apache.org<mailto:u...@tez.apache.org>" mailto:u...@tez.apache.org>> Subject: Hive on tez error I

Hive on tez error

I am trying to connect hive database(execution.engine value changed to tez) using Java code... In case of select query its working But in the case of INSERT getting an error: The error looks like. Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoo

Re: Running hive on tez locally

> How do I point Hive to tez libraries? Is it sufficient to populate >CLASSPATH environment variables with location of tez libraries ? The easiest option there is to populate the hive-config.sh with these two parameters https://github.com/t3rmin4t0r/tez-autobuild/blob/master/Makefile#L138 Once

Re: Running hive on tez locally

t: Re: Running hive on tez locally Thanks for the configs. When I run hive it crashes because TEZ libraries were not found. How do I point Hive to tez libraries? Is it sufficient to populate CLASSPATH environment variables with location of tez libraries ? Raajay On Fri, Aug 7, 2015 at 3:

Re: Running hive on tez locally

set tez.staging-dir=/tmp; > set tez.ignore.lib.uris=true; > set tez.runtime.optimize.local.fetch=true; > set tez.local.mode=true; > set hive.execution.engine=tez;​ > > > ​ > > > > -- > *From:* Raajay > *Sent:* Friday, August 07, 201

Re: Running hive on tez locally

; set tez.ignore.lib.uris=true; set tez.runtime.optimize.local.fetch=true; set tez.local.mode=true; set hive.execution.engine=tez;? ? From: Raajay Sent: Friday, August 07, 2015 1:06 PM To: user@hive.apache.org Subject: Running hive on tez locally I have been running

Re: Running hive on tez locally

EZ jobs even though > "hive.execution.engine=tez" is set. > > Is that expected ? If yes, what is the ideal environment for debugging hive > on tez? > > Raajay

Running hive on tez locally

I have been running Hive queries on a single node (no HDFS). I realize that the queries get compiled as map-reduce jobs and not as TEZ jobs even though "hive.execution.engine=tez" is set. Is that expected ? If yes, what is the ideal environment for debugging hive on tez? Raajay

Re: Hive on Tez much slower than MR

Hey Jörn, thanks for the response! Unfortunately I'm kinda stuck on the version I am. We do plan on moving to ORC at some point. I need to dig more into the implementation of how Vectorized execution works. The documentation ( https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execu

Re: Hive on Tez much slower than MR

Always use the newest version of Hive. You should use orc or parquet wherever possible. If you use orc then you should explicitly enable storage indexes and insert your table sorted (eg for the query below you would sort on x). Additionally you should enable statistics. Compression may bring addit

Re: Hive on Tez much slower than MR

I was able to bring the performance in line with MR by enabling reduce side vectorization, which apparently wasn't enabled in my cluster. The documentation regarding this is odd as it says ORC is required, but none of my tables are using ORC. > On Aug 5, 2015, at 3:48 PM, William Slacum wro

Hive on Tez much slower than MR

Hi all, I'm using Hive 0.14, Tez 0.5.2, and Hadoop 2.6.0. I have a very simple query of the form `select count(*) from my_table where x > 0 and x < 1500`. The table has ~50 columns in it and not all are populated. My total dataset size is ~20TB. When I run with MapReduce, I can generally see a m

Re: execution error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.teztask error on hive on tez

.org>> Date: Wednesday, July 29, 2015 at 5:54 PM To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>>, "u...@tez.apache.org<mailto:u...@tez.apache.org>" mailto:u...@tez.apache.org>> Subject: execution error, return

execution error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.teztask error on hive on tez

iam using hive 1.0 and tez 0.7 whenever iam performing insert query its returns following error: execution error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.teztask

how is the hive application name generated (for Hive on Tez)?

When a Hive job is submitted via Templeton/WebHCat, the server will launch a MR application named "TempletonControllerJob" which will launch the "real" Hive jobs. We want to know the exact mapping relationship between the "real" Hive on Tez application and the Temp

RE: Hive on Tez query failed with ³wrong key class"

...@hortonworks.com] On Behalf Of Gopal Vijayaraghavan Sent: Monday, July 27, 2015 9:20 PM To: u...@tez.apache.org; user@hive.apache.org Cc: Jim Green Subject: Re: Hive on Tez query failed with ³wrong key class" > From the java code which creates the sequence file, it has set the key &

Re: Hive on Tez query failed with ³wrong key class"

> From the java code which creates the sequence file, it has set the key >class to NullWritable.class: > job.setOutputKeyClass(org.apache.hadoop.io.NullWritable.class); ... > I think that caused the mismatch: > wrong key class: org.apache.hadoop.io.BytesWritable is not class >org.apache.hadoop.i

Re: Hive on Tez query failed with “wrong key class"

Next(HiveContextAwareRecordReader.java:358) > ... 21 more > ], > > > > On Tue, Jul 21, 2015 at 11:26 AM, Bikas Saha > wrote: > >> A full stack trace would help determine is this is a Tez issue or hive >> issue. >> >> >> >> *

Re: Hive on Tez query failed with “wrong key class"

uly 21, 2015 11:12 AM > *To:* u...@tez.apache.org; user@hive.apache.org > *Subject:* Hive on Tez query failed with “wrong key class" > > > > Hi Team, > > > > Env: Hive 1.0 on Tez 0.5.3 > > Query is a simple group-by on top of sequence table. > > > &

  1   2   >