Hi Ismael, Thanks for the valuable feedback. 1. No docker image specific release process: This was one of our considered approaches, but we thought that docker image shouldn't block AK release. Though I agree, treating docker image as another artifact for every AK release makes much more sense. Hence, releasing a new version of Kafka for the affected branch in such a scenario is a much cleaner approach. Added this as the accepted approach in the KIP. 2. EOL policy: Updated in the KIP.
Thanks and regards, Vedarth On Sun, Oct 22, 2023 at 11:20 PM Ismael Juma <m...@ismaeljuma.com> wrote: > Hi Vedarth, > > I think we shouldn't introduce any new release process that is docker > specific. We should consider the software in the docker image in the same > way as consider third party dependencies today - if there is a high > severity CVE affecting any of them, we aim to release a new version of > Kafka for the affected branch. It would include the latest Kafka code from > the branch. > > Additionally, we should specify the EOL policy in this KIP - we are not > changing it as part of it. One interesting detail is that the release > document claims we support the last 3 releases, but the reality has been a > bit different - we tend to support the 2 most recent releases unless it's a > high severity CVE in Kafka itself (these tend to be much rarer, > thankfully). > > Ismael > > On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma <vedarth.sha...@gmail.com> > wrote: > > > Hi Mickael, > > Thanks for going through the KIP and providing valuable feedback. > > > > 1. We will support the latest LTS version of Java supported by Apache > > Kafka. > > 2. We will provide support for the last three releases. We've added a > > detailed example of this in the KIP under our EOL policy. > > 3. We can establish a nightly cron job using GitHub Actions and > leverage > > an open-source vulnerability scanning tool like trivy ( > > https://github.com/aquasecurity/trivy), to get vulnerability reports > on > > all supported images. This tool offers a straightforward way to > > integrate > > vulnerability checks directly into our GitHub Actions workflow. > > 4. That's a good suggestion to have a GitHub Actions workflow. We will > > implement a GitHub Actions workflow to automate the build and testing > > process. > > 5. Regarding the release process, we observed that there isn't an > > existing CI/CD pipeline. We can consider the addition of a GitHub > > workflow > > to facilitate the release process. > > > > Please let us know your thoughts on the above. > > > > Thanks and regards, > > Vedarth > > > > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison <mickael.mai...@gmail.com > > > > wrote: > > > > > Hi Krishna, > > > > > > Overall I'm supportive of having an official docker image. > > > I have a few questions: > > > - Can you clarify the process of selecting the Java version? Is the > > > proposal to only pick LTS versions? or to pick the highest version > > > supported by Kafka? > > > - Once a new Kafka version is released, what happens to the image > > > containing the previous release? Do we expect to still update it in > > > case of CVEs? If so for how long? > > > - How will we get notified that the base image has a CVE? > > > - Rather than having scripts PMC members have to run from their > > > machines, would it e possible to have a Jenkins job or GitHub action? > > > > > > Thanks, > > > Mickael > > > > > > > > > > > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma > > > <vedarth.sha...@gmail.com> wrote: > > > > > > > > Hi Manikumar, > > > > > > > > Thanks for the feedback! > > > > > > > > 1. We propose the addition of a new directory named "docker" at the > > root > > > of > > > > the repository, where all Docker-related code will be stored. A > > detailed > > > > directory structure has been added in the KIP. > > > > 2. We request the creation of an Apache Kafka repository > (apache/kafka) > > > on > > > > DockerHub, to be administered under the The Apache Software > Foundation > > > > <https://hub.docker.com/u/apache>. The PMC members should have the > > > > necessary permissions for pushing updates to the docker repo. > > > > > > > > Thanks and regards, > > > > Vedarth > > > > > > > > > > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar <manikumar.re...@gmail.com > > > > > wrote: > > > > > > > > > Hi Krishna, Vedarth, > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > 1. Can we add directory structure of Docker Image related files in > > > Kafka > > > > > repo. > > > > > > > > > > 2. > Steps for the Docker image release will be included in the > > Release > > > > > Process doc of Apache Kafka > > > > > > > > > > Can we list down the requirements (repos, accounts) for releasing > > > images to > > > > > docker hub. I am mainly asking because PMC needs to request docker > > hub > > > > > access/repos. > > > > > I can help in getting required repos/accounts. > > > > > https://infra.apache.org/docker-hub-policy.html > > > > > > > > > > > > > > > Thanks, > > > > > Manikumar > > > > > > > > > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal < > > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > > > Hi Viktor, > > > > > > > > > > > > I've noticed there are two types of custom jar configurations: > > > > > > > > > > > > 1. *Type 1*: In this case, only the class name is required(e.g > > > > > > *authorizer.class.name > > > > > > <http://authorizer.class.name>**)* This can be configured by > > the > > > > > > following steps: > > > > > > - Mount the jar in the container. > > > > > > - Configure the *CLASSPATH* environment variable (used by > > > > > > *kafka-run-class.sh*) by providing the mounted path to it. > > > This can > > > > > > be passed as an environment variable to the docker > container. > > > > > > 2. *Type 2*: Here, in addition to the class name, classpath > can > > > also > > > > > be > > > > > > configured (eg *remote.log.metadata.manager.class.name > > > > > > <http://remote.log.metadata.manager.class.name> *and > > > > > > *remote.log.metadata.manager.class.path*). This can be > > configured > > > by > > > > > the > > > > > > following steps: > > > > > > - Mount the jar in the container. > > > > > > - Configure the respective *class.path* property. > > > > > > > > > > > > Regards, > > > > > > Krishna > > > > > > > > > > > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal < > > > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > > > > > Hi Viktor, > > > > > > > Thanks for the questions. > > > > > > > > > > > > > > 1. While the docker image outlined in KIP-975 is designed > for > > > > > > > production environments, it is equally suitable for > > development > > > and > > > > > > testing > > > > > > > purposes. We will furnish the docker image, allowing users > the > > > > > > flexibility > > > > > > > to employ it according to their specific needs. > > > > > > > 2. The configs will be injected into the docker container > > > through > > > > > > > environment variables. These environment variables will > have a > > > > > prefix > > > > > > > allowing for efficient parsing to extract the relevant > > > > > > properties.(Will add > > > > > > > this implementation in the KIP as well once we converge on > > > this.) > > > > > > > 3. Regarding this question, I'll conduct a test on my end > > after > > > > > > > gaining a better understanding, and then provide you with a > > > > > response. > > > > > > > > > > > > > > Regards, > > > > > > > Krishna > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass > > > > > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > > > > > > > >> Hi Ismael, > > > > > > >> > > > > > > >> I'm not trying to advocate against the docker image, I just > > > pointed > > > > > out > > > > > > >> that the current scoping of the KIP may be a bit too generic > and > > > > > thought > > > > > > >> that KIP-974 and KIP-975 were aiming for mostly the same thing > > > and can > > > > > > be > > > > > > >> discussed under one umbrella. Apologies if this was rooted in > a > > > > > > >> misunderstanding. > > > > > > >> > > > > > > >> Kirshna, > > > > > > >> > > > > > > >> I think we need to refine the KIP a bit more. I think there > are > > > some > > > > > > >> interfaces that we need to include in the KIP as Kafka has > > > plugins in > > > > > > >> certain cases where users are expected to provide > implementation > > > and I > > > > > > >> think it's worth discussing this in the KIP as they're kind of > > > > > > interfaces > > > > > > >> for users. Here are my questions in order: > > > > > > >> 1. In what environments do you want the image to be used? As I > > > > > > understand > > > > > > >> it would replace the current testing image and serve as a > basis > > > for > > > > > > >> development, but would it aim at production use cases too > > > > > > (docker-compose, > > > > > > >> Kubernetes, etc.)? > > > > > > >> 2. How do you plan to forward configs to the broker? Do we > > expect > > > a > > > > > > >> populated server.properties file placed in a certain location > or > > > > > should > > > > > > >> the > > > > > > >> docker image create this file based on some input (like env > > vars)? > > > > > > >> 3. Certain parts can be pluggable, like metric reporters or > > > remote log > > > > > > >> implementations that were just introduced by KIP-405. These > > > manifest > > > > > in > > > > > > >> jar > > > > > > >> files that must be put on the classpath of Kafka while certain > > > > > > classnames > > > > > > >> have to be configured. How do you plan to implement this, how > do > > > we > > > > > > >> allow users to configure such things? > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Viktor > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole > > > > > > >> <kevers...@cloudflare.com.invalid> wrote: > > > > > > >> > > > > > > >> > Hello, > > > > > > >> > > > > > > > >> > I think this would be a wonderful improvement to the > > ecosystem. > > > > > While > > > > > > >> > Viktor is correct that most Docker pipelines eventually lead > > to > > > a > > > > > > >> > kubernetes deployment, that should not stop us from creating > > an > > > > > > >> > Official Docker Image. Creating a Docker image would allow > us > > to > > > > > > ensure > > > > > > >> a > > > > > > >> > level of quality and support for people who want to deploy > > > Kafka as > > > > > a > > > > > > >> > container on baremetal machines, it could allow us to create > > > > > > >> > a sandbox/developer environment for new contributors and > > > developers > > > > > to > > > > > > >> test > > > > > > >> > and have a single agreed upon environment that kafka works > in > > > for > > > > > > future > > > > > > >> > KIPs and would most likely spawn more contributions from > > people > > > > > > wanting > > > > > > >> to > > > > > > >> > optimize kafka for k8s. > > > > > > >> > > > > > > > >> > > > > > > > >> > I am 100% for this and will gladly help if approved. > > > > > > >> > > > > > > > >> > Kenneth > > > > > > >> > > > > > > > >> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma < > > m...@ismaeljuma.com> > > > > > > wrote: > > > > > > >> > > > > > > > >> > > Hi Viktor, > > > > > > >> > > > > > > > > >> > > I disagree. Docker is a very popular deployment tool and > > it's > > > not > > > > > > only > > > > > > >> > used > > > > > > >> > > with Kubernetes. > > > > > > >> > > > > > > > > >> > > Ismael > > > > > > >> > > > > > > > > >> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass > > > > > > >> > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > >> > > > > > > > > >> > > > Hi Krishna, > > > > > > >> > > > > > > > > > >> > > > I think you should merge this KIP and KIP-974 > > > > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974> > > as > > > > > there > > > > > > >> are > > > > > > >> > overlaps as > > > > > > >> > > > Federico pointed out on KIP-974 > > > > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974 > >. > > I > > > > > think > > > > > > >> you > > > > > > >> > should keep that one as it > > > > > > >> > > > has well defined goals (improve tests) while I feel this > > > one is > > > > > > too > > > > > > >> > > > generic. Docker is usually just a tool for either > testing > > or > > > > > > >> > Kubernetes, > > > > > > >> > > so > > > > > > >> > > > they have very well defined use-cases. In the case of > > Flink > > > for > > > > > > >> > instance > > > > > > >> > > > the image is used for its kubernetes operator. The use > > case > > > > > would > > > > > > >> > > determine > > > > > > >> > > > a lot of things and I think a generic image would likely > > > not fit > > > > > > the > > > > > > >> > > needs > > > > > > >> > > > of all use-cases. > > > > > > >> > > > > > > > > > >> > > > Best, > > > > > > >> > > > Viktor > > > > > > >> > > > > > > > > > >> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal < > > > > > > >> > > > krishna0608agar...@gmail.com> > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Hi, > > > > > > >> > > > > Apache Kafka does not have an official docker image > > > currently. > > > > > > >> > > > > I want to submit a KIP to publish a docker image for > > > Apache > > > > > > Kafka. > > > > > > >> > > > > > > > > > > >> > > > > KIP-975 < > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975>: > > > > > > >> > Docker Image for Apache Kafka > > > > > > >> > > > > < > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > Regards, > > > > > > >> > > > > Krishna > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >
