Re: Flaky build in GitHub Actions

Thank you, Hyukjin! Dongjoon. On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon wrote: > I filed a ticket at GitHub. I will share more details when I get a > response from them. > > 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon 님이 작성: > >> Hi all, >> >> Looks like there's something going on in the machines

Fwd: Unpacking and using external modules with PySpark inside k8s

Hi, I am aware that some fellow members in this dev group were involved in creating scripts for running spark on kubernetes # To build additional PySpark docker image$ ./bin/docker-image-tool.sh -r -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build The problem I have

Re: Flaky build in GitHub Actions

I noticed that the worker decommissioning suite maybe seems to be running up against the memory limits so I'm going to try and see if I can get our memory usage down a bit as well while we wait for GH response. In the meantime, I'm assuming if things pass Jenkins we are OK with merging yes? On Wed

Re: Unpacking and using external modules with PySpark inside k8s

I managed to sort this one out. Please see https://stackoverflow.com/questions/68461865/unpacking-and-using-external-modules-with-pyspark-inside-kubernetes/68476548#68476548 HTH view my Linkedin profile *Disclaimer:* Use it at yo

Re: Flaky build in GitHub Actions

FYI, @Liang-Chi Hsieh is trying to control the memory in the test base at https://github.com/apache/spark/pull/33447 which looks almost promising now. While I don't object to merge things, would need to closely track how these tests go at Github Actions in his PR (and in the main Apache repo) 202

Time to start publishing Spark Docker Images?

Hi Folks, Many other distributed computing (https://hub.docker.com/r/rayproject/ray https://hub.docker.com/u/daskdev) and ASF projects ( https://hub.docker.com/u/apache) now publish their images to dockerhub. We've already got the docker image tooling in place, I think we'd need to ask the ASF to

Re:Time to start publishing Spark Docker Images?

+1Bests, Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant