Re: Publishing official docker images for KubernetesSchedulerBackend

I've been spending some time learning more about GPL, as it may apply to container images. Briefly, I believe that certain limits to the default "viral" nature of GPL apply in the case of a hypothetical Apache Spark image. For one example, the "mere aggregation"

Re: Publishing official docker images for KubernetesSchedulerBackend

I'd follow LEGAL-270, yes. The best resource on licensing is https://www.apache.org/legal/resolved.html ; it doesn't all have to be AL2, but needs to be compatible (sometimes with additional conditions). Auditing is basically entrusted to the PMC when voting on releases. I'll look at it with you.

Re: Publishing official docker images for KubernetesSchedulerBackend

Reasoning by analogy to other Apache projects is generally not sufficient when it come to securing legally permissible form or behavior -- that another project is doing something is not a guarantee that they are doing it right. If we have issues or legal questions, we need to formulate them and our

Re: Publishing official docker images for KubernetesSchedulerBackend

Unfortunately you'll need to chase down the license of all the bits that are distributed directly by the project. This was a big job back in the day for the Maven artifacts and some work to maintain. Most of the work is one-time, at least. On Tue, Dec 19, 2017 at 12:53 PM Erik Erlandson wrote: >

Re: Publishing official docker images for KubernetesSchedulerBackend

Agreed that the GPL family would be "toxic." The current images have been at least informally confirmed to use licenses that are ASF compatible. Is there an officially sanctioned method of license auditing that can be applied here? On Tue, Dec 19, 2017 at 11:45 AM, Sean Owen wrote: > I think t

Re: Publishing official docker images for KubernetesSchedulerBackend

I think that's all correct, though the license of third party dependencies is actually a difficult and sticky part. The ASF couldn't make a software release including any GPL software for example, and it's not just a matter of adding a disclaimer. Any actual bits distributed by the PMC would have t

Re: Publishing official docker images for KubernetesSchedulerBackend

I've been looking a bit more into ASF legal posture on licensing and container images. What I have found indicates that ASF considers container images to be just another variety of distribution channel. As such, it is acceptable to publish official releases; for example an image such as spark:v2.3

Re: Publishing official docker images for KubernetesSchedulerBackend

Currently the containers are based off alpine, which pulls in BSD2 and MIT licensing: https://github.com/apache/spark/pull/19717#discussion_r154502824 to the best of my understanding, neither of those poses a problem. If we based the image off of centos I'd also expect the licensing of any image

Re: Publishing official docker images for KubernetesSchedulerBackend

What licensing issues come into play? On Thu, Dec 14, 2017 at 4:00 PM, Erik Erlandson wrote: > We've been discussing the topic of container images a bit more. The > kubernetes back-end operates by executing some specific CMD and ENTRYPOINT > logic, which is different than mesos, and which is pr

Re: Publishing official docker images for KubernetesSchedulerBackend

We've been discussing the topic of container images a bit more. The kubernetes back-end operates by executing some specific CMD and ENTRYPOINT logic, which is different than mesos, and which is probably not practical to unify at this level. However: These CMD and ENTRYPOINT configurations are ess

Re: Publishing official docker images for KubernetesSchedulerBackend

We do support running on Apache Mesos via docker images - so this would not be restricted to k8s. But unlike mesos support, which has other modes of running, I believe k8s support more heavily depends on availability of docker images. Regards, Mridul On Wed, Nov 29, 2017 at 8:56 AM, Sean Owen

Re: Publishing official docker images for KubernetesSchedulerBackend

Would it be logical to provide Docker-based distributions of other pieces of Spark? or is this specific to K8S? The problem is we wouldn't generally also provide a distribution of Spark for the reasons you give, because if that, then why not RPMs and so on. On Wed, Nov 29, 2017 at 10:41 AM Anirudh

Re: Publishing official docker images for KubernetesSchedulerBackend

In this context, I think the docker images are similar to the binaries rather than an extension. It's packaging the compiled distribution to save people the effort of building one themselves, akin to binaries or the python package. For reference, this is the base dockerfile

Re: Publishing official docker images for KubernetesSchedulerBackend

It's probably also worth considering whether there is only one, well-defined, correct way to create such an image or whether this is a reasonable avenue for customization. Part of why we don't do something like maintain and publish canonical Debian packages for Spark is because different organizati

Re: Publishing official docker images for KubernetesSchedulerBackend

Source code is the primary release; compiled binary releases are conveniences that are also released. A docker image sounds fairly different though. To the extent it's the standard delivery mechanism for some artifact (think: pyspark on PyPI as well) that makes sense, but is that the situation? if