Re: map side join

Great info, thanks. Makes sense on the partition since those files can be shipped by themselves. These are "reference" tables, but one happens to be pretty long. Thanks, Abe On Thu, Apr 30, 2015 at 12:54 PM, Gopal Vijayaraghavan wrote: > Hi, > > > Using CDH 5.3 - Hive 0.13. Does a view help

Re: map side join

Hi, > Using CDH 5.3 - Hive 0.13. Does a view help here? Does how i format >the table help in reducing size? No, a view does not help - they are not materialized and you need hive-1.0 to have temporary table support. The only way out is if you only have 1 filter column in the system. I assume

Re: map side join

Using CDH 5.3 - Hive 0.13. Does a view help here? Does how i format the table help in reducing size? Abe On Thu, Apr 30, 2015 at 11:07 AM, Gopal Vijayaraghavan wrote: > Hi, > > > its submitting the whole table to the job. if I use a view with the > >filter > > baked in, will that help? I do

Re: map side join

Hi, > its submitting the whole table to the job. if I use a view with the >filter > baked in, will that help? I don't want to have to jack up the JVM for >the > client/HiveServer2 to accommodate the full table. Which hive version are you using? If you¹re on a recent version like hive-1.0, this

map side join

Hi, I am doing a few map side joins in one query to load an user facing ORC table in order to denormalize. Two of the tables I am joining too are pretty large. I am setting hive.auto.convert.join.noconditionaltask.size pretty high. However, the join it self filters on those two tables, but it se

map-side join fails when a serialized table contains arrays

Hi, I got the attached error on a map-side join where a serialized table contains an array column. When setting map-side join off via setting hive.mapjoin.optimized.hashtable=false, exceptions do not occur. It seems that a wrong ObjectInspector was set at CommonJoinOperator#initializeOp. I am

Map side join failed when setting hive.optimize.cp to false

Hive version 0.12.0 To enable Map side join, we enable: set hive.auto.convert.join=true; set hive.auto.convert.join.noconditionaltask = true; set hive.auto.convert.join.noconditionaltask.size = 12800; However, when we also set: hive.optimize.cp=false, Map side join will fail with

Re: Issue while inserting data in the hive table using map side join

Hi Anirudh, Below are some links depicting the problem MIGHT BE related to data nodes. Please go thru the same and let us know if it was useful. 1. http://hansmire.tumblr.com 2. http://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo Hive Experts- Kindly share your suggestions/findings on the sa

Issue while inserting data in the hive table using map side join

up in the map side join. This is a 20 node cluster with 8 GB ram on each machine. -- thanks Anirudh Visit me at : www.anirudhkala.in

Re: Map-side join memory limit is too low

Searching the JIRA for HADOOP_HEAPSIZE turned up this new ticket (and related ones mentioned in the comments): HADOOP-10245 : The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java > with "-Xmx" options twice. The impact is that

Re: Map-side join memory limit is too low

try "set hive.mapred.local.mem=7000" or add it to hive-site.xml instead of modifying hive-env.sh HADOOP_HEAPSIZE is not in use. Should fix documentation of it. Thanks, Navis 2014-01-31 Avrilia Floratou : > Hi, > I'm running hive 0.12 on yarn and I'm trying to convert a common join into > a map

Map-side join memory limit is too low

Hi, I'm running hive 0.12 on yarn and I'm trying to convert a common join into a map join. My map join fails and from the logs I can see that the memory limit is very low: Starting to launch local task to process map join; maximum memory = 514523136 How can I increase the maximum memory? I'

ClassCastException during reduce-side join, but not map-side join

1 STRING, ... c5 STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION 's3n://t1/'; CREATE EXTERNAL TABLE t2 ( c1 STRING, ... c18 STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION 's3n://t2/'; Finally, changing Q1 above to use a map-sid

Re: Map side join

Hi, To conclude this thread I am summarizing my experiences. Correct me if think // observed otherwise. 1) For Map side join you need to set the flag hive.auto.convert.join=true; Map side join works well with multiple table and multiple Join condition. 2) You can change the size of the small

Re: map side join with group by

n >> >> Yeah. My original question is that is there a way to force Hive (or >> rather to say, is it possible) to execute map side join at mapper phase and >> group by in reduce phase. So instead of launching a map only job (join) and >> map reduce job (group by), doing it

Re: map side join with group by

provide more ideas On Fri, Dec 14, 2012 at 12:42 AM, Chen Song wrote: > Nitin > > Yeah. My original question is that is there a way to force Hive (or rather > to say, is it possible) to execute map side join at mapper phase and group > by in reduce phase. So instead of launching a map

Re: map side join with group by

Nitin Yeah. My original question is that is there a way to force Hive (or rather to say, is it possible) to execute map side join at mapper phase and group by in reduce phase. So instead of launching a map only job (join) and map reduce job (group by), doing it altogether in a single MR job. This

Re: map side join with group by

keys are different). But for map side join, the joins would be > complete by the end of the map phase, and outputs should be ready to be > distributed to reducers based on group by keys. > > Chen > > > On Thu, Dec 13, 2012 at 11:04 AM, Nitin Pawar wrote: > >> Thats beca

Re: Map side join

KS > > Sent from remote device, Please excuse typos > -- > *From: * Souvik Banerjee > *Date: *Thu, 13 Dec 2012 12:00:16 -0600 > *To: *; > *Subject: *Re: Map side join > > Hi Bejoy, > > The input files are non-compressed text file. > There

Re: map side join with group by

Understood that fact that it is impossible in the same MR job if both join and group by are gonna happen in the reduce phase (because the join keys and group by keys are different). But for map side join, the joins would be complete by the end of the map phase, and outputs should be ready to be

Re: Map side join

from remote device, Please excuse typos -Original Message- From: Souvik Banerjee Date: Thu, 13 Dec 2012 12:00:16 To: ; Subject: Re: Map side join Hi Bejoy, The input files are non-compressed text file. There are enough free slots in the cluster. Can you please let me know can I increase

Re: Map side join

e > *Date: *Wed, 12 Dec 2012 14:27:27 -0600 > *To: *; > *ReplyTo: * user@hive.apache.org > *Subject: *Re: Map side join > > Hi Bejoy, > > Yes I ran the pi example. It was fine. > Regarding the HIVE Job what I found is that it took 4 hrs for the first > map job to get

Re: Map side join

Banerjee Date: Wed, 12 Dec 2012 14:27:27 To: ; Reply-To: user@hive.apache.org Subject: Re: Map side join Hi Bejoy, Yes I ran the pi example. It was fine. Regarding the HIVE Job what I found is that it took 4 hrs for the first map job to get completed. Those map tasks were doing their job and

Re: map side join with group by

n plans and the Semantic Analyzer code. >>> >>> And for completeness, there is a conditional task (starting Hive 0.7) >>> that will convert your joins automatically to map joins where >>> applicable. This can be enabled by enabling hive.auto.convert.join >>> pro

Re: map side join with group by

ert your joins automatically to map joins where >> applicable. This can be enabled by enabling hive.auto.convert.join >> property. >> >> Mark >> >> On Wed, Dec 12, 2012 at 3:32 PM, Chen Song >> wrote: >> > I have a silly question on how Hive inter

Re: map side join with group by

> > And for completeness, there is a conditional task (starting Hive 0.7) > that will convert your joins automatically to map joins where > applicable. This can be enabled by enabling hive.auto.convert.join > property. > > Mark > > On Wed, Dec 12, 2012 at 3:32 PM, Chen Son

Re: map side join with group by

ling hive.auto.convert.join property. Mark On Wed, Dec 12, 2012 at 3:32 PM, Chen Song wrote: > I have a silly question on how Hive interpretes a simple query with both map > side join and group by. > > Below query will translate into two jobs, with the 1st one as a map only job >

map side join with group by

I have a silly question on how Hive interpretes a simple query with both map side join and group by. Below query will translate into two jobs, with the 1st one as a map only job doing the join and storing the output in a intermediary location, and the 2nd one as a map-reduce job taking the output

Re: Map side join

g skeptical in > task, Tasktracker or jobtracker logs? > > > Regards > Bejoy KS > > Sent from remote device, Please excuse typos > -- > *From: * Souvik Banerjee > *Date: *Tue, 11 Dec 2012 17:12:20 -0600 > *To: *; > *ReplyTo: * user

Re: Map side join

-Original Message- From: Souvik Banerjee Date: Tue, 11 Dec 2012 17:12:20 To: ; Reply-To: user@hive.apache.org Subject: Re: Map side join Hello Everybody, Need help in for on HIVE join. As we were talking about the Map side join I tried that. I set the flag set hive.auto.convert.join=true; I

Re: Map side join

Hello Everybody, Need help in for on HIVE join. As we were talking about the Map side join I tried that. I set the flag set hive.auto.convert.join=true; I saw Hive converts the same to map join while launching the job. But the problem is that none of the map job progresses in my case. I made the

Re: Map side join

Hi Bejoy, That's wonderful. Thanks for your reply. What I was wondering if HIVE can do map side join with more than one condition on JOIN clause. I'll simply try it out and post the result. Thanks once again. Regards, Souvik. On Fri, Dec 7, 2012 at 2:10 PM, wrote: > ** >

Re: Map side join

Subject: Map side join Hello everybody, I have got a question. I didn't came across any post which says somethign about this. I have got two tables. Lets say A and B. I want to join A & B in HIVE. I am currently using HIVE 0.9 version. The join would be on few columns. like on (A.id1 = B

Map side join

.id2) AND (A.id3 = B.id3) Can I ask HIVE to use map side join in this scenario? Should I give a hint to HIVE by saying /*+mapjoin(B)*/ Get back to me if you want any more information in this regard. Thanks and regards, Souvik.

Re: Map side join and Serde jar in distributed cache missing

s >> table >> > needs a custom serde, which is added every time using add Jars(size of >> jar >> > is 25KB) in hive. >> > The problem is when hive performs map side join, classes in that serde >> jar >> > is not loaded and class not found excep

Re: Map side join and Serde jar in distributed cache missing

ek Pratap Singh > wrote: > > Hi all, > > > > I have enabled automatic Map join for any table less than 50MB. This > table > > needs a custom serde, which is added every time using add Jars(size of > jar > > is 25KB) in hive. > > The problem is when hive pe

Re: Map side join and Serde jar in distributed cache missing

0MB. This table > needs a custom serde, which is added every time using add Jars(size of jar > is 25KB) in hive. > The problem is when hive performs map side join, classes in that serde jar > is not loaded and class not found exception is thrown. But if I disable map > side join, it wo

Map side join and Serde jar in distributed cache missing

Hi all, I have enabled automatic Map join for any table less than 50MB. This table needs a custom serde, which is added every time using add Jars(size of jar is 25KB) in hive. The problem is when hive performs map side join, classes in that serde jar is not loaded and class not found exception is

Re: Map side join

task, > as will the order of the join condition. In this example, large.key was > always on the left side of the join conditions. > > > Matt Tucker > > -Original Message- > From: Abhishek [mailto:abhishek.dod...@gmail.com] > Sent: Wednesday, June 13, 2012 11:13

RE: Map side join

join conditions. Matt Tucker -Original Message- From: Abhishek [mailto:abhishek.dod...@gmail.com] Sent: Wednesday, June 13, 2012 11:13 AM To: user@hive.apache.org Subject: Map side join Hi all, How map side join in hive, can be used to join multiple tables(suppose 5 tables). Regards

Map side join

Hi all, How map side join in hive, can be used to join multiple tables(suppose 5 tables). Regards Abhishek Sent from my iPhone

Re: Hive map side join with Distributed cache

x27;d common-user@ and CC'd you. On Mon, Jun 11, 2012 at 9:46 PM, abhishek dodda wrote: > hi all, > > Map side join with distributed cache how to do this? can any one help > me on this. > > Regards > Abhishek. -- Harsh J