Thanks for the KIP Vince and Farid! Glad to see Bazel being considered for Kafka and really impressive work on the demo repository!
From the benchmarks, I'd imagine the build would only get faster once BUILD.bazel files don't glob the entire source/test set for a module but are more fine grained. I suppose it would be even faster if we were to use RBE [0] for executing builds/tests in parallel with caching [1]. Would be nice to get a sense of the potential improvements there if you've already done some experiments and some possible RBE options that we may use? Maybe another motivation/optimisation would be the use of target-determinator [2], previously discussed in [3] for running partial tests for PRs? I see that you mention potential changes in IDE configurations. It would be nice to mention plugins for common IDEs that support Bazel. IIRC, IntelliJ has decent Bazel support [4] but (IMHO) it's not as good as the Gradle integration. Can you perhaps discuss the potential changes in developer workflows, e.g. the need to add dependencies/files to BUILD.bazel files manually or running Gazelle to create BUILD.bazel files automatically? I see that in the example repository you declare the versions manually [5]. I think it would be cumbersome to maintain two separate versions list and perhaps we can generate the version mapping from `gradle/dependencies.gradle`, unless they need to deviate for some reason? To add to some of @Chia's questions and perhaps answer them partially: > CP0: Can Bazel help simplify build.gradle (currently around 3826 lines)? I think some of the complexity moves to *.bzl, *.bazel files [6] [7] [8] The configuration is in a language called Starlark [9] which is a subset of Python that avoids non-determinism. I'd say the configuration is more verbose than Gradle but is easier to reason about. I often find Gradle plugins do a lot of magic and are harder to navigate. > CP1: Gradle commands are part of the public API, so in which Kafka version > should we announce the deprecation of Gradle commands? My impression is once we get rid of Gradle so Phase 5? > CP2: The example (https://github.com/confluentinc/kafka-bazel) seems to > modify the folder structure of the entire Kafka project. Is this change > required? My concern is that such a major change should be done in a major > release. If restructuring is a requirement for Bazel, does that mean we have > to accept it after this KIP pass? I stumbled upon Farid's blog post [10] which talks a bit about this and my impression is the overlay can be merged into the main repository without changing its structure, potentially in a side directory as LLVM did [11]. I suppose this is what Phase 2 is about? That said, I do think refining the build graph, which includes having more fine grained BUILD.bazel files, may warrant moving some classes/packages around. This is because Bazel isn't happy with cycles in the build graph while Gradle allows them within a module. I'm looking forward to trying the overlay repository in the coming days and will follow up with more questions/feedback! Regards, Gaurav [0]: https://bazel.build/remote/rbe [1]: https://bazel.build/remote/caching [2]: https://github.com/bazel-contrib/target-determinator [3]: https://lists.apache.org/thread/xqbj7ywmwf5zfycgo1ckyhl9zs43h1dd [4]: https://github.com/bazelbuild/intellij [5]: https://github.com/confluentinc/kafka-bazel/blob/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/dependencies/dependencies.bzl#L24 [6]: https://github.com/confluentinc/kafka-bazel/tree/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/tools-bazel [7]: https://github.com/confluentinc/kafka-bazel/tree/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/dependencies [8]: https://github.com/confluentinc/kafka-bazel/blob/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/WORKSPACE.bazel [9]: https://github.com/bazelbuild/starlark [10]: https://fzakaria.com/2024/08/29/bazel-overlay-pattern.html [11]: https://github.com/llvm/llvm-project/tree/main/utils/bazel > On 22 Nov 2024, at 03:39, Chia-Ping Tsai <chia7...@apache.org> wrote: > > hi Vince > > Thanks for introducing this interesting KIP! I'm not an expert in Bazel, so > please be patient with my potentially odd questions: > > CP0: Can Bazel help simplify build.gradle (currently around 3826 lines)? > > CP1: Gradle commands are part of the public API, so in which Kafka version > should we announce the deprecation of Gradle commands? > > CP2: The example (https://github.com/confluentinc/kafka-bazel) seems to > modify the folder structure of the entire Kafka project. Is this change > required? My concern is that such a major change should be done in a major > release. If restructuring is a requirement for Bazel, does that mean we have > to accept it after this KIP pass? > > Best, > Chia-Ping > > On 2024/11/21 21:16:50 Vince Rose wrote: >> Hey devs, >> >> I want to start a discussion about introducing Bazel as a build system for >> Apache Kafka. >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1115%3A+Bazel+Builds >> >> -Vince >>
