That configuration looks bad. With only two cores in use and 1GB used by
the app. Few points-
1. Please oversubscribe those CPUs to at-least twice the amount of cores
you have to start-with and then tune if it freezes
2. Allocate all of the CPU cores and memory to your running app (I assume
it is
We were getting the same problem also. Funny thing our code worked with
larger data set and failed for a reduced data set. Anyway we are thinking on
passing stacksize override params to jvm may be that can help you.
Please give it a try and let me know.
--conf spark.executor.extraJavaOptions=-Xs
ala program -- perhaps we should move it to the spark
> > config file.
> >
> > Driver memory and executor memory are set to 12GB
> > parallelism is set to 8
> > Kryo serializer is used
> > Number of retainedJobs and retainedStages has been increased to check
>
ontext would be of interest here?
>
> Regards,
> Raghava.
>
> On Sun, Apr 17, 2016 at 10:54 PM, Anuj Kumar wrote:
>
>> If the data file is same then it should have similar distribution of
>> keys. Few queries-
>>
>> 1. Did you compare the number of partition
If the data file is same then it should have similar distribution of keys.
Few queries-
1. Did you compare the number of partitions in both the cases?
2. Did you compare the resource allocation for Spark Shell vs Scala Program
being submitted?
Also, can you please share the details of Spark Conte
#other-configuration-options
but I have not tried this configuration parameter earlier.
OTOH, I didn't understand the motive of the query. What exactly is the
purpose? - Are you looking for distinct guids?
Regards,
Anuj
On Tue, Oct 6, 2015 at 3:42 AM, Muhammad Ahsan
wrote:
> Hello
