If I launch more executors, GC gets worse.
2015-02-06 10:47 GMT+01:00 Guillermo Ortiz <konstt2...@gmail.com>:
> This is an execution with 80 executors
>
> MetricMin25th percentileMedian75th percentileMax
> Duration 31s 44s 50s 1.1min 2.6 min
> GC Time 70ms 0.1s 0.3s 4s 53 s
> Input 128.0MB 128.0MB 128.0MB 128.0MB 128.0MB
>
> I executed as well with 40 executors
> MetricMin25th percentileMedian75th percentileMax
> Duration 26s 28s 28s 30s 35s
> GC Time 54ms 60ms 66ms 80ms 0.4 s
> Input 128.0MB 128.0MB 128.0MB 128.0MB 128.0 MB
>
> I checked the %iowait and %steal in a worker it's all right in both of them
> I understand the value of yarn.nodemanager.resource.memory-mb is for
> each worker in the cluster and not the total value for YARN. it's
> configured at 196GB right now. (I have 5 workers)
> 80executors x 4Gb = 320Gb, it shouldn't be a problem.
>
>
> 2015-02-06 10:03 GMT+01:00 Sandy Ryza <sandy.r...@cloudera.com>:
>> Yes, having many more cores than disks and all writing at the same time can
>> definitely cause performance issues. Though that wouldn't explain the high
>> GC. What percent of task time does the web UI report that tasks are
>> spending in GC?
>>
>> On Fri, Feb 6, 2015 at 12:56 AM, Guillermo Ortiz <konstt2...@gmail.com>
>> wrote:
>>>
>>> Yes, It's surpressing to me as well....
>>>
>>> I tried to execute it with different configurations,
>>>
>>> sudo -u hdfs spark-submit --master yarn-client --class
>>> com.mycompany.app.App --num-executors 40 --executor-memory 4g
>>> Example-1.0-SNAPSHOT.jar hdfs://ip:8020/tmp/sparkTest/ file22.bin
>>> parameters
>>>
>>> This is what I executed with different values in num-executors and
>>> executor-memory.
>>> What do you think there are too many executors for those HDDs? Could
>>> it be the reason because of each executor takes more time?
>>>
>>> 2015-02-06 9:36 GMT+01:00 Sandy Ryza <sandy.r...@cloudera.com>:
>>> > That's definitely surprising to me that you would be hitting a lot of GC
>>> > for
>>> > this scenario. Are you setting --executor-cores and --executor-memory?
>>> > What are you setting them to?
>>> >
>>> > -Sandy
>>> >
>>> > On Thu, Feb 5, 2015 at 10:17 AM, Guillermo Ortiz <konstt2...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Any idea why if I use more containers I get a lot of stopped because
>>> >> GC?
>>> >>
>>> >> 2015-02-05 8:59 GMT+01:00 Guillermo Ortiz <konstt2...@gmail.com>:
>>> >> > I'm not caching the data. with "each iteration I mean,, each 128mb
>>> >> > that a executor has to process.
>>> >> >
>>> >> > The code is pretty simple.
>>> >> >
>>> >> > final Conversor c = new Conversor(null, null, null,
>>> >> > longFields,typeFields);
>>> >> > SparkConf conf = new SparkConf().setAppName("Simple Application");
>>> >> > JavaSparkContext sc = new JavaSparkContext(conf);
>>> >> > JavaRDD<byte[]> rdd = sc.binaryRecords(path, c.calculaLongBlock());
>>> >> >
>>> >> > JavaRDD<String> rddString = rdd.map(new Function<byte[], String>() {
>>> >> > @Override
>>> >> > public String call(byte[] arg0) throws Exception {
>>> >> > String result = c.parse(arg0).toString();
>>> >> > return result;
>>> >> > }
>>> >> > });
>>> >> > rddString.saveAsTextFile(url + "/output/" +
>>> >> > System.currentTimeMillis()+
>>> >> > "/");
>>> >> >
>>> >> > The parse function just takes an array of bytes and applies some
>>> >> > transformations like,,,
>>> >> > [0..3] an integer, [4...20] an String, [21..27] another String and so
>>> >> > on.
>>> >> >
>>> >> > It's just a test code, I'd like to understand what it's happeing.
>>> >> >
>>> >> > 2015-02-04 18:57 GMT+01:00 Sandy Ryza <sandy.r...@cloudera.com>:
>>> >> >> Hi Guillermo,
>>> >> >>
>>> >> >> What exactly do you mean by "each iteration"? Are you caching data
>>> >> >> in
>>> >> >> memory?
>>> >> >>
>>> >> >> -Sandy
>>> >> >>
>>> >> >> On Wed, Feb 4, 2015 at 5:02 AM, Guillermo Ortiz
>>> >> >> <konstt2...@gmail.com>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> I execute a job in Spark where I'm processing a file of 80Gb in
>>> >> >>> HDFS.
>>> >> >>> I have 5 slaves:
>>> >> >>> (32cores /256Gb / 7physical disks) x 5
>>> >> >>>
>>> >> >>> I have been trying many different configurations with YARN.
>>> >> >>> yarn.nodemanager.resource.memory-mb 196Gb
>>> >> >>> yarn.nodemanager.resource.cpu-vcores 24
>>> >> >>>
>>> >> >>> I have tried to execute the job with different number of executors
>>> >> >>> a
>>> >> >>> memory (1-4g)
>>> >> >>> With 20 executors takes 25s each iteration (128mb) and it never has
>>> >> >>> a
>>> >> >>> really long time waiting because GC.
>>> >> >>>
>>> >> >>> When I execute around 60 executors the process time it's about 45s
>>> >> >>> and
>>> >> >>> some tasks take until one minute because GC.
>>> >> >>>
>>> >> >>> I have no idea why it's calling GC when I execute more executors
>>> >> >>> simultaneously.
>>> >> >>> The another question it's why it takes more time to execute each
>>> >> >>> block. My theory about the this it's because there're only 7
>>> >> >>> physical
>>> >> >>> disks and it's not the same 5 processes writing than 20.
>>> >> >>>
>>> >> >>> The code is pretty simple, it's just a map function which parse a
>>> >> >>> line
>>> >> >>> and write the output in HDFS. There're a lot of substrings inside
>>> >> >>> of
>>> >> >>> the function what it could cause GC.
>>> >> >>>
>>> >> >>> Any theory about?
>>> >> >>>
>>> >> >>>
>>> >> >>> ---------------------------------------------------------------------
>>> >> >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> >> >>> For additional commands, e-mail: user-h...@spark.apache.org
>>> >> >>>
>>> >> >>
>>> >
>>> >
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
