FYI - filed bunch of issues for flaky tests in recent CI builds

Hi devs, I've found bunch of test failures (intermittently) in both CI build for master branch as well as PR builder (only checked for mine) yesterday. I just filed issues which I observed, but I guess there's more as I only checked my PR. https://issues.apache.org/jira/browse/SPARK-29129 https:/

Re: Welcoming some new committers and PMC members

Congrats to all! On Tue, Sep 17, 2019 at 6:51 PM Bryan Cutler wrote: > Congratulations, all well deserved! > > On Thu, Sep 12, 2019, 3:32 AM Jacek Laskowski wrote: > >> Hi, >> >> What a great news! Congrats to all awarded and the community for voting >> them in! >> >> p.s. I think it should go

Re: Welcoming some new committers and PMC members

Congratulations, all well deserved! On Thu, Sep 12, 2019, 3:32 AM Jacek Laskowski wrote: > Hi, > > What a great news! Congrats to all awarded and the community for voting > them in! > > p.s. I think it should go to the user mailing list too. > > Pozdrawiam, > Jacek Laskowski > > https://abo

Re: [build system] weird mvn errors post-cache cleaning

Oh, thank you for fixing that! :) Bests, Dongjoon. On Tue, Sep 17, 2019 at 12:57 PM Shane Knapp wrote: > > ah, i found this sucker on amp-jenkins-worker-02: > > s/02/06 > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apach

Re: [build system] weird mvn errors post-cache cleaning

> ah, i found this sucker on amp-jenkins-worker-02: s/02/06 - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [build system] weird mvn errors post-cache cleaning

ah, i found this sucker on amp-jenkins-worker-02: -bash-4.1$ cat /home/jenkins/.m2/repository/org/apache/spark/spark-parent_2.12/maven-metadata-local.xml org.apache.spark spark-parent_2.12 spark-155410 spark-155410 20190917185916 tUpdated> <--- not goo

Re: [build system] weird mvn errors post-cache cleaning

that's what i literally just did! i wiped the .m2, .ivy and per-executor sbt caches on all of these machines. maybe it was just a network burp. also, it's only 2 builds: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110809/console https://amplab.cs.berkeley.edu/jenkins/job/S

Re: [build system] weird mvn errors post-cache cleaning

That's super weird; can you just delete ~/.m2 and let it download the internet again? or at least blow away the downloaded Kafka dir? Turning it on and off, so to speak, often works. On Tue, Sep 17, 2019 at 2:41 PM Shane Knapp wrote: > > a bunch of the PRB builds are now failing w/various permuta

[build system] weird mvn errors post-cache cleaning

a bunch of the PRB builds are now failing w/various permutations of the following: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install (default-cli) on project spark-sql-kafka-0-10_2.12: ArtifactInstallerException: Failed to install metadata org.apache.spa

Re: [build system] short (~1hr max) downtime

this is done and jenkins is building again! On Tue, Sep 17, 2019 at 10:14 AM Shane Knapp wrote: > > i'm going to clean up the spark workspaces on the jenkins workers and > clear out ivy and maven caches. > > this means no new builds will be started as of right now, and current > builds cancelled.

Re: Thoughts on Spark 3 release, or a preview release

I don’t know if it will be feasible to merge all of SPARK-25299 into Spark 3. There are a number of APIs that will be submitted for review, and I wouldn’t want to block the release on negotiating these changes, as the decisions we make for each API can be pretty involved. Our original plan w

[build system] short (~1hr max) downtime

i'm going to clean up the spark workspaces on the jenkins workers and clear out ivy and maven caches. this means no new builds will be started as of right now, and current builds cancelled. i'll wait a little bit for some nearly-complete PRB builds to finish, and i will restart any other builds t

Re: Thoughts on Spark 3 release, or a preview release

https://issues.apache.org/jira/browse/SPARK-28264 SPARK-28264 Revisiting Python / pandas UDF sounds critical for 3.0 preview Xiao On Mon, Sep 16, 2019 at 12:22 PM Erik Erlandson wrote: > > I'm in favor of adding SPARK-25299 > - Use remote stor