Re: Zeppelin out of memory issue - (GC overhead limit exceeded)

[Eric Charles]

You also have to check the memory you give to the spark driver 
(spark.driver.memory property)

On 26/03/17 07:40, RUSHIKESH RAUT wrote:
Yes I know it inevitable if the data is large. I want to know how do I
increase the interpreter memory to handle large data?
Thanks,
Rushikesh Raut

On Mar 26, 2017 8:56 AM, "Jianfeng (Jeff) Zhang" <jzh...@hortonworks.com
<mailto:jzh...@hortonworks.com>> wrote:


    How large is your data ? This problem is inevitable if your data is
    too large, you can try to use spark data frame if that works for you.





    Best Regard,
    Jeff Zhang


    From: RUSHIKESH RAUT <rushikeshraut...@gmail.com
    <mailto:rushikeshraut...@gmail.com>>
    Reply-To: "users@zeppelin.apache.org
    <mailto:users@zeppelin.apache.org>" <users@zeppelin.apache.org
    <mailto:users@zeppelin.apache.org>>
    Date: Saturday, March 25, 2017 at 5:06 PM
    To: "users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>"
    <users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>>
    Subject: Zeppelin out of memory issue - (GC overhead limit exceeded)

    Hi everyone,

    I am trying to load some data from hive table into my notebook and
    then convert this dataframe into r dataframe using spark.r
    interpreter. This works perfectly for small amount of data.
    But if the data is increased then it gives me error

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    I have tried increasing the ZEPPELIN_MEM and ZEPPELIN_INTP_MEM in
    the zeppelin-env.cmd file but i am still facing this issue. I have
    used the following configuration

    set ZEPPELIN_MEM="-Xms4096m -Xmx4096m -XX:MaxPermSize=2048m"
    set ZEPPELIN_INTP_MEM="-Xmx4096m -Xms4096m -XX:MaxPermSize=2048m"

    I am sure that this much size should be sufficient for my data but
    still i am getting this same error. Any guidance will be much
    appreciated.

    Thanks,
    Rushikesh Raut