Re:Time to start publishing Spark Docker Images?

+1Bests, Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant 

Time to start publishing Spark Docker Images?

Hi Folks, Many other distributed computing (https://hub.docker.com/r/rayproject/ray https://hub.docker.com/u/daskdev) and ASF projects ( https://hub.docker.com/u/apache) now publish their images to dockerhub. We've already got the docker image tooling in place, I think we'd need to ask the ASF to

Re: Flaky build in GitHub Actions

FYI, @Liang-Chi Hsieh is trying to control the memory in the test base at https://github.com/apache/spark/pull/33447 which looks almost promising now. While I don't object to merge things, would need to closely track how these tests go at Github Actions in his PR (and in the main Apache repo) 202

Re: Unpacking and using external modules with PySpark inside k8s

I managed to sort this one out. Please see https://stackoverflow.com/questions/68461865/unpacking-and-using-external-modules-with-pyspark-inside-kubernetes/68476548#68476548 HTH view my Linkedin profile *Disclaimer:* Use it at yo

Re: Flaky build in GitHub Actions

I noticed that the worker decommissioning suite maybe seems to be running up against the memory limits so I'm going to try and see if I can get our memory usage down a bit as well while we wait for GH response. In the meantime, I'm assuming if things pass Jenkins we are OK with merging yes? On Wed

Fwd: Unpacking and using external modules with PySpark inside k8s

Hi, I am aware that some fellow members in this dev group were involved in creating scripts for running spark on kubernetes # To build additional PySpark docker image$ ./bin/docker-image-tool.sh -r -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build The problem I have

Re: Flaky build in GitHub Actions

Thank you, Hyukjin! Dongjoon. On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon wrote: > I filed a ticket at GitHub. I will share more details when I get a > response from them. > > 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon 님이 작성: > >> Hi all, >> >> Looks like there's something going on in the machines