Yeah the color on this is that 'snapshot' or 'nightly' builds are not quite _discouraged_ by the ASF, but need to be something only devs are likely to find and clearly signposted, because they aren't official blessed releases. It gets into a gray area if the project is 'officially' hosting a way to get snapshot builds. It is not at all impossible, just something that's come up and generated some angst in the past, so we dropped it.
On Thu, Jan 23, 2020 at 1:09 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > > Hi, Jim. > > Thank you for the proposal. I understand the request. > However, the following key benefit sounds like unofficial snapshot binary > releases. > > > For example, this was used to build a version of spark that included > > SPARK-28938 which has yet to be released and was necessary for > > spark-operator to work properly with GKE service accounts > > Historically, we removed the existing snapshot binaries in some personal > repositories and there is no plan to add it back. > Also, for snapshot dev jars, we use only the official Apache Maven snapshot > repository. > > For official releases, we aim to release Apache Spark source code (and its > artifacts) according to the pre-defined release cadence in an official manner. > > BTW, SPARK-28938 doesn't mean that we need to publish a docker image. Even in > the official release, as you know, we only provide a reference Dockerfile. > That's the reason why we don't publish docker image via GitHub Action (as of > Today). > > To achieve the following custom requirement, I'd like to recommend you to > have your own Dockerfile. > That is the best way for you to have the flexibility. > > > One value of this is the ability to create versions of dependent packages > > such as spark-on-k8s-operator > > Thanks, > Dongjoon. > > > On Thu, Jan 23, 2020 at 9:32 AM Jim Kleckner <j...@cloudphysics.com> wrote: >> >> This story [1] proposes adding a .gitlab-ci.yml file to make it easy to >> create artifacts and images for spark. >> >> Using this mechanism, people can submit any subsequent version of spark for >> building and image hosting with gitlab.com. >> >> There is a companion WIP branch [2] with a candidate and example for doing >> this. >> The exact steps for building are in the yml file [3]. >> The images get published into the namespace of the user as here [4] >> >> One value of this is the ability to create versions of dependent packages >> such as spark-on-k8s-operator that might use upgraded packages or >> modifications for testing. For example, this was used to build a version of >> spark that included SPARK-28938 which has yet to be released and was >> necessary for spark-operator to work properly with GKE service accounts [5]. >> >> Comments about desirability? >> >> [1] https://issues.apache.org/jira/browse/SPARK-30275 >> [2] https://gitlab.com/jkleckner/spark/tree/add-gitlab-ci-yml >> [3] https://gitlab.com/jkleckner/spark/blob/add-gitlab-ci-yml/.gitlab-ci.yml >> [4] https://gitlab.com/jkleckner/spark/container_registry >> [5] https://gitlab.com/jkleckner/spark-on-k8s-operator/container_registry --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
