The original PR from Liye didn't include test which exercises Kryo buffer size configured in mb which is below 2GB.
In my PR, I added such a test and it passed on Jenkins: https://github.com/apache/spark/pull/6390 FYI On Sun, May 24, 2015 at 8:08 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Please update to the following: > > commit c2f0821aad3b82dcd327e914c9b297e92526649d > Author: Zhang, Liye <liye.zh...@intel.com> > Date: Fri May 8 09:10:58 2015 +0100 > > [SPARK-7392] [CORE] bugfix: Kryo buffer size cannot be larger than 2M > > On Sun, May 24, 2015 at 8:04 AM, Debasish Das <debasish.da...@gmail.com> > wrote: > >> I am May 3rd commit: >> >> commit 49549d5a1a867c3ba25f5e4aec351d4102444bc0 >> >> Author: WangTaoTheTonic <wangtao...@huawei.com> >> >> Date: Sun May 3 00:47:47 2015 +0100 >> >> >> [SPARK-7031] [THRIFTSERVER] let thrift server take >> SPARK_DAEMON_MEMORY and SPARK_DAEMON_JAVA_OPTS >> >> On Sat, May 23, 2015 at 7:54 PM, Josh Rosen <rosenvi...@gmail.com> wrote: >> >>> Which commit of master are you building off? It looks like there was a >>> bugfix for an issue related to KryoSerializer buffer configuration: >>> https://github.com/apache/spark/pull/5934 >>> >>> That patch was committed two weeks ago, but you mentioned that you're >>> building off a newer version of master. Could you confirm the commit that >>> you're running? If this used to work but now throws an error, then this is >>> a regression that should be fixed; we shouldn't require you to perform a mb >>> -> kb conversion to work around this. >>> >>> On Sat, May 23, 2015 at 6:37 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Pardon me. >>>> >>>> Please use '8192k' >>>> >>>> Cheers >>>> >>>> On Sat, May 23, 2015 at 6:24 PM, Debasish Das <debasish.da...@gmail.com >>>> > wrote: >>>> >>>>> Tried "8mb"...still I am failing on the same error... >>>>> >>>>> On Sat, May 23, 2015 at 6:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>>> bq. it shuld be "8mb" >>>>>> >>>>>> Please use the above syntax. >>>>>> >>>>>> Cheers >>>>>> >>>>>> On Sat, May 23, 2015 at 6:04 PM, Debasish Das < >>>>>> debasish.da...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am on last week's master but all the examples that set up the >>>>>>> following >>>>>>> >>>>>>> .set("spark.kryoserializer.buffer", "8m") >>>>>>> >>>>>>> are failing with the following error: >>>>>>> >>>>>>> Exception in thread "main" java.lang.IllegalArgumentException: >>>>>>> spark.kryoserializer.buffer must be less than 2048 mb, got: + 8192 mb. >>>>>>> looks like buffer.mb is deprecated...Is "8m" is not the right syntax >>>>>>> to get 8mb kryo buffer or it shuld be "8mb" >>>>>>> >>>>>>> Thanks. >>>>>>> Deb >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
