Hi Raihan
The propery 'mapred.jobtracker.maxtasks.per.job' is a JobTracker level one and
not a task level one. Hence you cannot override it at task level. You need to
make modifications in mapred_site.xml also you may need to rebounce the JT as
well for the new value to come into effect.
Regards,
Bejoy KS
________________________________
From: Raihan Jamal <jamalrai...@gmail.com>
To: user@hive.apache.org
Sent: Thursday, October 4, 2012 5:24 AM
Subject: Re: org.apache.hadoop.ipc.RemoteException(java.io.IOException:
java.io.IOException
Just to add here
SojTimestampToDate will return data in this format only 2012/02/29 17:01:43
Raihan Jamal
On Wed, Oct 3, 2012 at 4:46 PM, Raihan Jamal <jamalrai...@gmail.com> wrote:
This is still not working as in the XML file the final property has been set as
true so that means I cannot override it.
>And this below simple query is also throwing same exception-
>
>
>SELECT event.app_payload ['n'] AS changed_cguid
>FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
>WHERE a.dt = '20120918'
>AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd
>HH:mm:ss') >= unix_timestamp('2012/09/18 00:00:00', 'yyyy/MM/dd HH:mm:ss')
>AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd
>HH:mm:ss') <= unix_timestamp('2012/09/18 02:00:00', 'yyyy/MM/dd HH:mm:ss')
>
>
>Exception I am getting:-
>
>
>Job Submission failed with exception
>'org.apache.hadoop.ipc.RemoteException(java.io.IOException:
>java.io.IOException: The number of tasks for this job 2070929 exceeds the
>configured limit 200000
>
>
>
>
>Any other suggestion what should I do to overcome this problem? May be any
>changes in the query can overcome this problem?
>
>
>Raihan Jamal
>
>
>
>On Wed, Oct 3, 2012 at 2:59 PM, Chalcy Raja <chalcy.r...@careerbuilder.com>
>wrote:
>
>Hi Raihan,
>>
>>You can set it in hive prompt like below,
>>set mapred.jobtracker.maxtasks.per.job=7777777;
>>
>>To see if it is set just type in hive prompt, set; and you’ll see this
>>parameter in the output.
>>
>>Hope this helps,
>>Chalcy
>>
>>From:Raihan Jamal [mailto:jamalrai...@gmail.com]
>>Sent: Wednesday, October 03, 2012 5:51 PM
>>To: user@hive.apache.org
>>Subject: Re: org.apache.hadoop.ipc.RemoteException(java.io.IOException:
>>java.io.IOException
>>
>>Ok. Found the issue I guess.
>>
>>This is the below settings we have in the mapred-site.xml for the
>>site-specific configuration in Hadoop. And that is the reason exception is
>>getting thrown.
>>
>><property>
>> <!-- 10,000 is 100 tasks per node on a 100-node cluster -->
>> <name>mapred.jobtracker.maxtasks.per.job</name>
>> <value>200000</value>
>> <final>true</final>
>> </property>
>>
>>How can I override these changes manually from the Hive prompt? Any
>>suggestions?
>>
>>
>>
>>Raihan Jamal
>>
>>
>>
>>On Wed, Oct 3, 2012 at 2:19 PM, Raihan Jamal <jamalrai...@gmail.com> wrote:
>>Can anyone help me out here? What does the below error means? And this is the
>>query I am using-
>>
>>SELECT cguid,
>>event_item,
>>event_timestamp,
>>event_site_id
>>FROM (
>>SELECT event.app_payload ['n'] AS cguid,
>>event.app_payload ['itm'] AS event_item,
>>max(event.event_timestamp) AS event_timestamp,
>>event.site_id AS event_site_id
>>FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
>>WHERE a.dt = '20120917'
>>AND event.app_payload ['n'] IS NOT NULL
>>AND instr(event.app_payload ['itm'], '%') = 0
>>AND event.app_payload ['itm'] IS NOT NULL
>>AND (
>>event.page_type_id = '4340'
>>OR event.page_type_id = '2047675'
>>)
>>GROUP BY event.app_payload ['n'],
>>event.site_id,
>>event.app_payload ['itm']
>>ORDER BY cguid,
>>event_timestamp DESC
>>) m
>>LEFT OUTER JOIN (
>>SELECT event.app_payload ['n'] AS changed_cguid
>>FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
>>WHERE a.dt = '20120918'
>>AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd
>>HH:mm:ss') >= unix_timestamp('2012/09/18 00:00:00', 'yyyy/MM/dd HH:mm:ss')
>>AND unix_timestamp(SojTimestampToDate(event.event_timestamp), 'yyyy/MM/dd
>>HH:mm:ss') <= unix_timestamp('2012/09/18 02:00:00', 'yyyy/MM/dd HH:mm:ss')
>>) n ON m.cguid = n.changed_cguid
>>WHERE n.changed_cguid IS NULL
>>
>>
>>
>>Raihan Jamal
>>
>>
>>
>>On Wed, Oct 3, 2012 at 11:05 AM, Raihan Jamal <jamalrai...@gmail.com> wrote:
>>I am running a Hive query and I am getting this exception below-
>>
>>Job Submission failed with exception
>>'org.apache.hadoop.ipc.RemoteException(java.io.IOException:
>>java.io.IOException: The number of tasks for this job 2072020 exceeds the
>>configured limit 200000
>>
>>I am not sure what does this error means? Can anyone help me out here?
>>
>>
>>
>>Raihan Jamal
>>
>>
>>
>
