It won't be used by PySpark and SparkR, will it? On 23 August 2017 at 23:40, Michael Armbrust <mich...@databricks.com> wrote:
> I think we already set the number of partitions to 5 in tests > <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> > ? > > On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz < > mszymkiew...@gmail.com> wrote: > >> Hi, >> >> From my experience it is possible to cut quite a lot by reducing >> spark.sql.shuffle.partitions to some reasonable value (let's say >> comparable to the number of cores). 200 is a serious overkill for most of >> the test cases anyway. >> >> >> Best, >> Maciej >> >> >> >> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote: >> >>> +1 for any efforts to recover Jenkins! >>> >>> >>> >>> Thank you for the direction. >>> >>> >>> >>> Bests, >>> >>> Dongjoon. >>> >>> >>> >>> *From: *Reynold Xin <r...@databricks.com> >>> *Date: *Sunday, August 20, 2017 at 5:53 PM >>> *To: *Dong Joon Hyun <dh...@hortonworks.com> >>> *Cc: *"dev@spark.apache.org" <dev@spark.apache.org> >>> *Subject: *Re: Increase Timeout or optimize Spark UT? >>> >>> >>> >>> It seems like it's time to look into how to cut down some of the test >>> runtimes. Test runtimes will slowly go up given the way development >>> happens. 3 hr is already a very long time for tests to run. >>> >>> >>> >>> >>> >>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com> >>> wrote: >>> >>> Hi, All. >>> >>> >>> >>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) >>> has been hitting the build timeout. >>> >>> >>> >>> Please see the build time trend. >>> >>> >>> >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >>> >>> >>> >>> All recent 22 builds fail due to timeout directly/indirectly. The last >>> success (SBT with Hadoop-2.7) is 15th August. >>> >>> >>> >>> We may do the followings. >>> >>> >>> >>> 1. Increase Build Timeout (3 hr 30 min) >>> 2. Optimize UTs (Scala/Java/Python/UT) >>> >>> >>> >>> But, Option 1 will be the immediate solution for now . Could you update >>> the Jenkins setup? >>> >>> >>> >>> Bests, >>> >>> Dongjoon. >>> >>> >>> >> >> > -- Z poważaniem, Maciej Szymkiewicz
