Hi Krishna, Overall I'm supportive of having an official docker image. I have a few questions: - Can you clarify the process of selecting the Java version? Is the proposal to only pick LTS versions? or to pick the highest version supported by Kafka? - Once a new Kafka version is released, what happens to the image containing the previous release? Do we expect to still update it in case of CVEs? If so for how long? - How will we get notified that the base image has a CVE? - Rather than having scripts PMC members have to run from their machines, would it e possible to have a Jenkins job or GitHub action?
Thanks, Mickael On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma <vedarth.sha...@gmail.com> wrote: > > Hi Manikumar, > > Thanks for the feedback! > > 1. We propose the addition of a new directory named "docker" at the root of > the repository, where all Docker-related code will be stored. A detailed > directory structure has been added in the KIP. > 2. We request the creation of an Apache Kafka repository (apache/kafka) on > DockerHub, to be administered under the The Apache Software Foundation > <https://hub.docker.com/u/apache>. The PMC members should have the > necessary permissions for pushing updates to the docker repo. > > Thanks and regards, > Vedarth > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar <manikumar.re...@gmail.com> wrote: > > > Hi Krishna, Vedarth, > > > > Thanks for the KIP. > > > > 1. Can we add directory structure of Docker Image related files in Kafka > > repo. > > > > 2. > Steps for the Docker image release will be included in the Release > > Process doc of Apache Kafka > > > > Can we list down the requirements (repos, accounts) for releasing images to > > docker hub. I am mainly asking because PMC needs to request docker hub > > access/repos. > > I can help in getting required repos/accounts. > > https://infra.apache.org/docker-hub-policy.html > > > > > > Thanks, > > Manikumar > > > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal < > > krishna0608agar...@gmail.com> wrote: > > > > > Hi Viktor, > > > > > > I've noticed there are two types of custom jar configurations: > > > > > > 1. *Type 1*: In this case, only the class name is required(e.g > > > *authorizer.class.name > > > <http://authorizer.class.name>**)* This can be configured by the > > > following steps: > > > - Mount the jar in the container. > > > - Configure the *CLASSPATH* environment variable (used by > > > *kafka-run-class.sh*) by providing the mounted path to it. This can > > > be passed as an environment variable to the docker container. > > > 2. *Type 2*: Here, in addition to the class name, classpath can also > > be > > > configured (eg *remote.log.metadata.manager.class.name > > > <http://remote.log.metadata.manager.class.name> *and > > > *remote.log.metadata.manager.class.path*). This can be configured by > > the > > > following steps: > > > - Mount the jar in the container. > > > - Configure the respective *class.path* property. > > > > > > Regards, > > > Krishna > > > > > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal < > > > krishna0608agar...@gmail.com> wrote: > > > > > > > Hi Viktor, > > > > Thanks for the questions. > > > > > > > > 1. While the docker image outlined in KIP-975 is designed for > > > > production environments, it is equally suitable for development and > > > testing > > > > purposes. We will furnish the docker image, allowing users the > > > flexibility > > > > to employ it according to their specific needs. > > > > 2. The configs will be injected into the docker container through > > > > environment variables. These environment variables will have a > > prefix > > > > allowing for efficient parsing to extract the relevant > > > properties.(Will add > > > > this implementation in the KIP as well once we converge on this.) > > > > 3. Regarding this question, I'll conduct a test on my end after > > > > gaining a better understanding, and then provide you with a > > response. > > > > > > > > Regards, > > > > Krishna > > > > > > > > > > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass > > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > >> Hi Ismael, > > > >> > > > >> I'm not trying to advocate against the docker image, I just pointed > > out > > > >> that the current scoping of the KIP may be a bit too generic and > > thought > > > >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can > > > be > > > >> discussed under one umbrella. Apologies if this was rooted in a > > > >> misunderstanding. > > > >> > > > >> Kirshna, > > > >> > > > >> I think we need to refine the KIP a bit more. I think there are some > > > >> interfaces that we need to include in the KIP as Kafka has plugins in > > > >> certain cases where users are expected to provide implementation and I > > > >> think it's worth discussing this in the KIP as they're kind of > > > interfaces > > > >> for users. Here are my questions in order: > > > >> 1. In what environments do you want the image to be used? As I > > > understand > > > >> it would replace the current testing image and serve as a basis for > > > >> development, but would it aim at production use cases too > > > (docker-compose, > > > >> Kubernetes, etc.)? > > > >> 2. How do you plan to forward configs to the broker? Do we expect a > > > >> populated server.properties file placed in a certain location or > > should > > > >> the > > > >> docker image create this file based on some input (like env vars)? > > > >> 3. Certain parts can be pluggable, like metric reporters or remote log > > > >> implementations that were just introduced by KIP-405. These manifest > > in > > > >> jar > > > >> files that must be put on the classpath of Kafka while certain > > > classnames > > > >> have to be configured. How do you plan to implement this, how do we > > > >> allow users to configure such things? > > > >> > > > >> Thanks, > > > >> Viktor > > > >> > > > >> > > > >> > > > >> > > > >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole > > > >> <kevers...@cloudflare.com.invalid> wrote: > > > >> > > > >> > Hello, > > > >> > > > > >> > I think this would be a wonderful improvement to the ecosystem. > > While > > > >> > Viktor is correct that most Docker pipelines eventually lead to a > > > >> > kubernetes deployment, that should not stop us from creating an > > > >> > Official Docker Image. Creating a Docker image would allow us to > > > ensure > > > >> a > > > >> > level of quality and support for people who want to deploy Kafka as > > a > > > >> > container on baremetal machines, it could allow us to create > > > >> > a sandbox/developer environment for new contributors and developers > > to > > > >> test > > > >> > and have a single agreed upon environment that kafka works in for > > > future > > > >> > KIPs and would most likely spawn more contributions from people > > > wanting > > > >> to > > > >> > optimize kafka for k8s. > > > >> > > > > >> > > > > >> > I am 100% for this and will gladly help if approved. > > > >> > > > > >> > Kenneth > > > >> > > > > >> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma <m...@ismaeljuma.com> > > > wrote: > > > >> > > > > >> > > Hi Viktor, > > > >> > > > > > >> > > I disagree. Docker is a very popular deployment tool and it's not > > > only > > > >> > used > > > >> > > with Kubernetes. > > > >> > > > > > >> > > Ismael > > > >> > > > > > >> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass > > > >> > > <viktor.somo...@cloudera.com.invalid> wrote: > > > >> > > > > > >> > > > Hi Krishna, > > > >> > > > > > > >> > > > I think you should merge this KIP and KIP-974 > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974> as > > there > > > >> are > > > >> > overlaps as > > > >> > > > Federico pointed out on KIP-974 > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974>. I > > think > > > >> you > > > >> > should keep that one as it > > > >> > > > has well defined goals (improve tests) while I feel this one is > > > too > > > >> > > > generic. Docker is usually just a tool for either testing or > > > >> > Kubernetes, > > > >> > > so > > > >> > > > they have very well defined use-cases. In the case of Flink for > > > >> > instance > > > >> > > > the image is used for its kubernetes operator. The use case > > would > > > >> > > determine > > > >> > > > a lot of things and I think a generic image would likely not fit > > > the > > > >> > > needs > > > >> > > > of all use-cases. > > > >> > > > > > > >> > > > Best, > > > >> > > > Viktor > > > >> > > > > > > >> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal < > > > >> > > > krishna0608agar...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Hi, > > > >> > > > > Apache Kafka does not have an official docker image currently. > > > >> > > > > I want to submit a KIP to publish a docker image for Apache > > > Kafka. > > > >> > > > > > > > >> > > > > KIP-975 < > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975>: > > > >> > Docker Image for Apache Kafka > > > >> > > > > < > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka > > > >> > > > > > > > > >> > > > > > > > >> > > > > Regards, > > > >> > > > > Krishna > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >
