Re: ASF board report draft for August

Hi Matei, 3.2 will also include support for pushed based shuffle (spip SPARK-30602). Regards, Mridul On Mon, Aug 9, 2021 at 9:26 PM Hyukjin Kwon wrote: > > Are you referring to what version of Koala project? 1.8.1? > > Yes, the latest version 1.8.1. > > 2021년 8월 10일 (화) 오전 11:07, Igor Costa

Re: ASF board report draft for August

> Are you referring to what version of Koala project? 1.8.1? Yes, the latest version 1.8.1. 2021년 8월 10일 (화) 오전 11:07, Igor Costa 님이 작성: > Hi Matei, nice update > > > Just one question, when you mention “ We are working on Spark 3.2.0 as > our next release, with a release candidate likely to com

Re: ASF board report draft for August

There is an SPIP passed and ready for Spark 3.2: pandas API on Spark: - JIRA: SPIP: Support pandas API layer on PySpark ( https://issues.apache.org/jira/browse/SPARK-34849) - Vote: [VOTE] SPIP: Support pandas API layer on PySpark ( https://www.mail-archive.com/dev@spark.apache.org/msg27605.html) -

Re: ASF board report draft for August

Hi Matei, nice update Just one question, when you mention “ We are working on Spark 3.2.0 as our next release, with a release candidate likely to come soon. Spark 3.2 includes a new Pandas API for Apache Spark based on the Koalas project” Are you referring to what version of Koala project? 1.8.

ASF board report draft for August

It’s time for our quarterly report to the ASF board, which we need to send out this Wednesday. I wrote the draft below based on community activity — let me know if you’d like to add or change anything: == Description: Apache Spark is a fast and general engin

Re: [build system] half of the jenkins workers are down

Thank you, Shane! Xiao shane knapp ☠ 于2021年8月9日周一 下午1:26写道： > turns out that minikube/k8s and friends were being oom-killed and this was > causing all sorts of weirdnesses. > > i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and > we'll keep an eye on things and see how they g

Re: [build system] half of the jenkins workers are down

turns out that minikube/k8s and friends were being oom-killed and this was causing all sorts of weirdnesses. i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and we'll keep an eye on things and see how they go. On Mon, Aug 9, 2021 at 12:02 PM shane knapp ☠ wrote: > as workers a

Performance of PySpark jobs on the Kubernetes cluster

Hi, I have a basic question to ask. I am running a Google k8s cluster (AKA GKE) with three nodes each having configuration below e2-standard-2 (2 vCPUs, 8 GB memory) spark-submit is launched from another node (actually a data proc single node that I have just upgraded to e2-custom (4 vCPUs, 8

Re: [build system] half of the jenkins workers are down

as workers are continuing to fail, i've stopped jenkins from accepting new builds for the time being. more updates as they come. On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ wrote: > happy monday! > > the server gods did not smile upon us this weekend, and 4 of the workers > are down. we'll mo

[build system] half of the jenkins workers are down

happy monday! the server gods did not smile upon us this weekend, and 4 of the workers are down. we'll most likely need to head to our colo some time today and give them an in-person kick and see what's going on. i'll send an update when they're back up. shane -- Shane Knapp Computer Guy / Voi