Hi Warren,
it's often an exception stemming from an OOM at the executor level.
If you are caching data make sure you spill to disk, if needed.
You could also try to increase off-heap memory to alleviate the issue.
Of course also handing more memory to the executor helps.
Best regards,
Alessandr
Hi All,
I have seen this exception many times in my production environment for long
running batch job. Is there some stigmatization of all root causes of this
exception? Below is my analysis:
1. This happens when executor try to fetch MapStatus of some shuffle.
2. Each executor maintains a local
cutor-memory 3g --driver-memory 8g --conf
spark.executor.memoryOverhead=2g --conf spark.network.timeout=2000s --class
XXX --jars /path/to/jars /path/to/application
'''
And got the following errors:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an
d=0, mapId=-1,
> reduceId=86, message=
> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
> location for shuffle 0
> at org.apache.spark.MapOutputTracker$$anonfun$org$
> apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(
> MapOutputTracker.scala:5
=-1,
reduceId=86, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:542)
at
org.apache.spark.MapOutputTrac
:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:460)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache
mailto:veljko.skar...@gmail.com>> wrote:
Hi,
I keep getting some variation of the following error:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 2
Does anyone know what this might indicate? Is it a memory issue? Any general
guidance appreciated.
Hi,
I keep getting some variation of the following error:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 2
Does anyone know what this might indicate? Is it a memory issue? Any
general guidance appreciated.
76k+ rows. But heavily skewed in the middle (thus requires
>>>> repartitioning)
>>>> and each row has around 100k of data after serialization. The job
>>>> always got
>>>> stuck in repartitioning. Namely, the job will constantly get following
&g
t;> only
>>> 76k+ rows. But heavily skewed in the middle (thus requires
>>> repartitioning)
>>> and each row has around 100k of data after serialization. The job always
>>> got
>>> stuck in repartitioning. Namely, the job will constantly get following
>>
rors and retries:
>>
>> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
>> location for shuffle
>>
>> org.apache.spark.shuffle.FetchFailedException: Error in opening
>> FileSegmentManagedBuffer
>>
>> org.apache.spark.shuffle.Fetch
owing
> errors and retries:
>
> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
> location for shuffle
>
> org.apache.spark.shuffle.FetchFailedException: Error in opening
> FileSegmentManagedBuffer
>
> org.apache.spark.shuffle.FetchFailedException:
&
e job always got
stuck in repartitioning. Namely, the job will constantly get following
errors and retries:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle
org.apache.spark.shuffle.FetchFailedException: Error in opening
FileSegmentManagedB
have
>> got fix it .Please help me.
>> Here is the track.
>>
>> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
>> location for shuffle 0
>> at
>> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$
;> got fix it .Please help me.
>> Here is the track.
>>
>> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
>> location for shuffle 0
>> at
>> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$
FailedException: Missing an output
> location for shuffle 0
> at
> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:386)
> at
> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTrack
Again,when I do larger file Spark-sql query, error occured.Anyone have got
fix it .Please help me.
Here is the track.
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker
Hi Darin, you might increase spark.yarn.executor.memoryOverhead to see if
it fixes the problem. Please take a look of this report:
https://issues.apache.org/jira/browse/SPARK-4996
On Fri, Feb 27, 2015 at 12:38 AM, Arush Kharbanda <
ar...@sigmoidanalytics.com> wrote:
> Can you share what error you
Can you share what error you are getting when the job fails.
On Thu, Feb 26, 2015 at 4:32 AM, Darin McBeath
wrote:
> I'm using Spark 1.2, stand-alone cluster on ec2 I have a cluster of 8
> r3.8xlarge machines but limit the job to only 128 cores. I have also tried
> other things such as setting
I'm using Spark 1.2, stand-alone cluster on ec2 I have a cluster of 8
r3.8xlarge machines but limit the job to only 128 cores. I have also tried
other things such as setting 4 workers per r3.8xlarge and 67gb each but this
made no difference.
The job frequently fails at the end in this step (sa
20 matches
Mail list logo
