BTW, the situation seems to become worse, now we lost two builds. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/ - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/
Dongjoon. On Thu, Aug 24, 2017 at 2:27 AM, Maciej Szymkiewicz <mszymkiew...@gmail.com> wrote: > It won't be used by PySpark and SparkR, will it? > > On 23 August 2017 at 23:40, Michael Armbrust <mich...@databricks.com> > wrote: > >> I think we already set the number of partitions to 5 in tests >> <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> >> ? >> >> On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz < >> mszymkiew...@gmail.com> wrote: >> >>> Hi, >>> >>> From my experience it is possible to cut quite a lot by reducing >>> spark.sql.shuffle.partitions to some reasonable value (let's say >>> comparable to the number of cores). 200 is a serious overkill for most of >>> the test cases anyway. >>> >>> >>> Best, >>> Maciej >>> >>> >>> >>> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> >>> wrote: >>> >>>> +1 for any efforts to recover Jenkins! >>>> >>>> >>>> >>>> Thank you for the direction. >>>> >>>> >>>> >>>> Bests, >>>> >>>> Dongjoon. >>>> >>>> >>>> >>>> *From: *Reynold Xin <r...@databricks.com> >>>> *Date: *Sunday, August 20, 2017 at 5:53 PM >>>> *To: *Dong Joon Hyun <dh...@hortonworks.com> >>>> *Cc: *"dev@spark.apache.org" <dev@spark.apache.org> >>>> *Subject: *Re: Increase Timeout or optimize Spark UT? >>>> >>>> >>>> >>>> It seems like it's time to look into how to cut down some of the test >>>> runtimes. Test runtimes will slowly go up given the way development >>>> happens. 3 hr is already a very long time for tests to run. >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com> >>>> wrote: >>>> >>>> Hi, All. >>>> >>>> >>>> >>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) >>>> has been hitting the build timeout. >>>> >>>> >>>> >>>> Please see the build time trend. >>>> >>>> >>>> >>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >>>> >>>> >>>> >>>> All recent 22 builds fail due to timeout directly/indirectly. The last >>>> success (SBT with Hadoop-2.7) is 15th August. >>>> >>>> >>>> >>>> We may do the followings. >>>> >>>> >>>> >>>> 1. Increase Build Timeout (3 hr 30 min) >>>> 2. Optimize UTs (Scala/Java/Python/UT) >>>> >>>> >>>> >>>> But, Option 1 will be the immediate solution for now . Could you update >>>> the Jenkins setup? >>>> >>>> >>>> >>>> Bests, >>>> >>>> Dongjoon. >>>> >>>> >>>> >>> >>> >> > > > -- > > Z poważaniem, > Maciej Szymkiewicz > >
