Hi, Jim. Thank you for the proposal. I understand the request. However, the following key benefit sounds like unofficial snapshot binary releases.
> For example, this was used to build a version of spark that included SPARK-28938 which has yet to be released and was necessary for spark-operator to work properly with GKE service accounts Historically, we removed the existing snapshot binaries in some personal repositories and there is no plan to add it back. Also, for snapshot dev jars, we use only the official Apache Maven snapshot repository. For official releases, we aim to release Apache Spark source code (and its artifacts) according to the pre-defined release cadence in an official manner. BTW, SPARK-28938 doesn't mean that we need to publish a docker image. Even in the official release, as you know, we only provide a reference Dockerfile. That's the reason why we don't publish docker image via GitHub Action (as of Today). To achieve the following custom requirement, I'd like to recommend you to have your own Dockerfile. That is the best way for you to have the flexibility. > One value of this is the ability to create versions of dependent packages such as spark-on-k8s-operator Thanks, Dongjoon. On Thu, Jan 23, 2020 at 9:32 AM Jim Kleckner <j...@cloudphysics.com> wrote: > This story [1] proposes adding a .gitlab-ci.yml file to make it easy to > create artifacts and images for spark. > > Using this mechanism, people can submit any subsequent version of spark for > building and image hosting with gitlab.com. > > There is a companion WIP branch [2] with a candidate and example for doing > this. > The exact steps for building are in the yml file [3]. > The images get published into the namespace of the user as here [4] > > One value of this is the ability to create versions of dependent packages > such as spark-on-k8s-operator that might use upgraded packages or > modifications for testing. For example, this was used to build a version > of spark that included SPARK-28938 which has yet to be released and was > necessary for spark-operator to work properly with GKE service accounts > [5]. > > Comments about desirability? > > [1] https://issues.apache.org/jira/browse/SPARK-30275 > [2] https://gitlab.com/jkleckner/spark/tree/add-gitlab-ci-yml > [3] https://gitlab.com/jkleckner/spark > /blob/add-gitlab-ci-yml/.gitlab-ci.yml > [4] https://gitlab.com/jkleckner/spark/container_registry > [5] https://gitlab.com/jkleckner/spark-on-k8s-operator/container_registry >
