RE: org.apache.hadoop.ipc.RemoteException(java.io.IOException: java.io.IOException

[Chalcy Raja]

Hi Raihan,

You can set it in hive prompt like below,
set mapred.jobtracker.maxtasks.per.job=7777777;

To see if it is set just type in hive prompt, set;  and you'll see this 
parameter in the output.

Hope this helps,
Chalcy

From: Raihan Jamal [mailto:jamalrai...@gmail.com]
Sent: Wednesday, October 03, 2012 5:51 PM
To: user@hive.apache.org
Subject: Re: org.apache.hadoop.ipc.RemoteException(java.io.IOException: 
java.io.IOException

Ok. Found the issue I guess.

This is the below settings we have in the  mapred-site.xml for the 
site-specific configuration in Hadoop. And that is the reason exception is 
getting thrown.

<property>
    <!-- 10,000 is 100 tasks per node on a 100-node cluster -->
    <name>mapred.jobtracker.maxtasks.per.job</name>
    <value>200000</value>
    <final>true</final>
  </property>

How can I override these changes manually from the Hive prompt? Any suggestions?



Raihan Jamal


On Wed, Oct 3, 2012 at 2:19 PM, Raihan Jamal 
<jamalrai...@gmail.com<mailto:jamalrai...@gmail.com>> wrote:
Can anyone help me out here? What does the below error means? And this is the 
query I am using-

SELECT cguid,
event_item,
event_timestamp,
event_site_id
FROM (
SELECT event.app_payload ['n'] AS cguid,
event.app_payload ['itm'] AS event_item,
max(event.event_timestamp) AS event_timestamp,
event.site_id AS event_site_id
FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
WHERE a.dt = '20120917'
AND event.app_payload ['n'] IS NOT NULL
AND instr(event.app_payload ['itm'], '%') = 0
AND event.app_payload ['itm'] IS NOT NULL
AND (
event.page_type_id = '4340'
OR event.page_type_id = '2047675'
)
GROUP BY event.app_payload ['n'],
event.site_id,
event.app_payload ['itm']
ORDER BY cguid,
event_timestamp DESC
) m
LEFT OUTER JOIN (
SELECT event.app_payload ['n'] AS changed_cguid
FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
WHERE a.dt = '20120918'
AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd 
HH:mm:ss') >= unix_timestamp('2012/09/18 00:00:00', 'yyyy/MM/dd HH:mm:ss')
AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd 
HH:mm:ss') <= unix_timestamp('2012/09/18 02:00:00', 'yyyy/MM/dd HH:mm:ss')
) n ON m.cguid = n.changed_cguid
WHERE n.changed_cguid IS NULL



Raihan Jamal


On Wed, Oct 3, 2012 at 11:05 AM, Raihan Jamal 
<jamalrai...@gmail.com<mailto:jamalrai...@gmail.com>> wrote:
I am running a Hive query and I am getting this exception below-

Job Submission failed with exception 
'org.apache.hadoop.ipc.RemoteException(java.io.IOException: 
java.io.IOException: The number of tasks for this job 2072020 exceeds the 
configured limit 200000

I am not sure what does this error means? Can anyone help me out here?



Raihan Jamal