Regarding 2, relying on CI alone is quite challenging from a debugging perspective. Committers typically rely on the ability to run locally or via a VM on a given platform. For platforms that are not particularly popular, I am not sure if the Apache Kafka committer should incur the burden of supporting them.
Ismael On Thu, Mar 31, 2022 at 9:28 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Ismael, > > Thanks for the feedback! > > 1. I agree we need to run system tests. As far as I can tell we don't > currently run system tests for any platforms in the Apache CI, not > even for x86_64. System tests require a lot of resources so I'm not > sure we can run them daily on all platforms in the Apache CI. However, > I think we should be able to run them in the Apache CI (probably only > triggered by committers) whenever required, for example when doing a > release, or when debugging broken tests. We've not enabled integration > tests on aarch64 and pp64le yet so it's hard to gauge if a single > machine is enough to handle all builds. But if we also want to run > system tests regularly, ensuring we have at least 2 machines for each > platforms seems like a requirement. > > 2. Machines are managed by the Apache Infra team. I'd expect the > process to be similar for all platforms. If Infra can't manage some > machines effectively and help us debug issues on them, then they > should not be part of the CI. > > Thanks, > Mickael > > On Thu, Mar 31, 2022 at 5:50 PM Mickael Maison <mickael.mai...@gmail.com> > wrote: > > > > Hi Divij, > > > > Thanks for taking a look at the KIP! > > > > 1. I'm not sure we should include the kernel as part of the support > > statement. Kafka does not interact with the kernel directly and > > instead relies on the JVM to make the right system calls. So I think > > it's best to keep that out. > > > > 2. That's a good point. The system tests that Ismael mentioned include > > benchmarks, so if we run them we would get some performance metrics. > > > > Thanks, > > Mickael > > > > > > On Wed, Mar 30, 2022 at 3:10 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > Thanks for the KIP. I would say there are 2 requirements to fully > support > > > platforms that the KIP does not mention: > > > > > > 1. Running system tests on that platform nightly. > > > 2. Committers need a mechanism to be able to debug and fix issues > affecting > > > the platforms. > > > > > > arm64 seems like the kind of platform where we can achieve both, but > the > > > others are more challenging. > > > > > > Ismael > > > > > > On Wed, Mar 30, 2022 at 4:01 AM Mickael Maison < > mickael.mai...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > I recently started a thread > > > > (https://lists.apache.org/thread/4ffpspc59cnzqbgjf3l50ykcvyp7vwtv) > > > > about whether we should consider aarch64 (ARM64) and ppc64le > (PowerPC) > > > > as supported platforms because we've recently started using these > > > > platforms in the Kafka CI. > > > > > > > > Unfortunately nobody replied so I guess a better way to start a > > > > discussion is to make a proposal. I opened KIP-826 to propose clearly > > > > defining the platforms we support in our docs and also introducing a > > > > process for adding additional platforms in the future: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-826%3A+Define+platforms+supported > > > > > > > > Let me know if you have any feedback. > > > > > > > > Thanks, > > > > Mickael > > > > >
