Bringing fzaka...@confluent.io into the discussion. On Fri, Nov 22, 2024 at 10:01 AM Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> wrote:
> Hi Vincent > > I would also like to say that I'm not an expert in Bazel either. Generally > I think it's good to reduce build and test times in every infra. While > reading up on Bazel, it seems like it's most useful in monolithic code > bases, such as Confluent as you mentioned in the KIP (so it's not a > coincidence :) ). I have a few thoughts though: > > SVV1. That 175 vs 100 seconds in the no-cache seems to be a big difference. > There is a performance benchmark by Gradle ( > https://blog.gradle.org/gradle-vs-bazel-jvm) which might be biased from > their side, but seems to conclude different results. One thing they mention > in the article is that they had one BAZEL.build for a project, instead of > the package level granularity. In your provided example, you seem to have > it on a per module basis. Could this be the difference in performance? Have > you tried it out with different granularities for Bazel? > > SVV2. I can share the arguments in CP2 as well. Bazel would optimally need > a build file in every package, which you'll need to maintain manually. This > is an extra burden on the developer side. Based on the example you cited, > Gradle seems to have better JVM integration compared to Bazel as the latter > one tries to be a generic build tool and not tuned for the JVM. I think > that this could be a potential concern. > > SVV3. Before switching to a completely different build system, I think we > have to ask ourselves: did we exhaust any optimization in Gradle that can > be done, and is it really that slower to build in it? As the article above > mentioned, if Bazel build files are defined on package level, then the > package level cyclic dependencies can be eliminated. This, however, can be > done with Java's module system too, which would be the idiomatic Java way. > If this is done, perhaps it could speed up Gradle builds as well. > > Best, > Viktor > > On Fri, Nov 22, 2024 at 5:43 PM Gaurav Narula <ka...@gnarula.com> wrote: > > > Thanks for the KIP Vince and Farid! > > > > Glad to see Bazel being considered for Kafka and really impressive work > on > > the demo repository! > > > > From the benchmarks, I'd imagine the build would only get faster once > > BUILD.bazel files don't glob the entire source/test set for a module but > > are more fine grained. I suppose it would be even faster if we were to > use > > RBE [0] for executing builds/tests in parallel with caching [1]. Would be > > nice to get a sense of the potential improvements there if you've already > > done some experiments and some possible RBE options that we may use? > Maybe > > another motivation/optimisation would be the use of target-determinator > > [2], previously discussed in [3] for running partial tests for PRs? > > > > I see that you mention potential changes in IDE configurations. It would > > be nice to mention plugins for common IDEs that support Bazel. IIRC, > > IntelliJ has decent Bazel support [4] but (IMHO) it's not as good as the > > Gradle integration. Can you perhaps discuss the potential changes in > > developer workflows, e.g. the need to add dependencies/files to > BUILD.bazel > > files manually or running Gazelle to create BUILD.bazel files > automatically? > > > > I see that in the example repository you declare the versions manually > > [5]. I think it would be cumbersome to maintain two separate versions > list > > and perhaps we can generate the version mapping from > > `gradle/dependencies.gradle`, unless they need to deviate for some > reason? > > > > To add to some of @Chia's questions and perhaps answer them partially: > > > > > CP0: Can Bazel help simplify build.gradle (currently around 3826 > lines)? > > > > I think some of the complexity moves to *.bzl, *.bazel files [6] [7] [8] > > > > The configuration is in a language called Starlark [9] which is a subset > > of Python that avoids non-determinism. I'd say the configuration is more > > verbose than Gradle but is easier to reason about. I often find Gradle > > plugins do a lot of magic and are harder to navigate. > > > > > CP1: Gradle commands are part of the public API, so in which Kafka > > version should we announce the deprecation of Gradle commands? > > > > My impression is once we get rid of Gradle so Phase 5? > > > > > CP2: The example (https://github.com/confluentinc/kafka-bazel) seems > to > > modify the folder structure of the entire Kafka project. Is this change > > required? My concern is that such a major change should be done in a > major > > release. If restructuring is a requirement for Bazel, does that mean we > > have to accept it after this KIP pass? > > > > I stumbled upon Farid's blog post [10] which talks a bit about this and > my > > impression is the overlay can be merged into the main repository without > > changing its structure, potentially in a side directory as LLVM did > [11]. I > > suppose this is what Phase 2 is about? > > > > That said, I do think refining the build graph, which includes having > more > > fine grained BUILD.bazel files, may warrant moving some classes/packages > > around. This is because Bazel isn't happy with cycles in the build graph > > while Gradle allows them within a module. > > > > I'm looking forward to trying the overlay repository in the coming days > > and will follow up with more questions/feedback! > > > > Regards, > > Gaurav > > > > [0]: https://bazel.build/remote/rbe > > [1]: https://bazel.build/remote/caching > > [2]: https://github.com/bazel-contrib/target-determinator > > [3]: https://lists.apache.org/thread/xqbj7ywmwf5zfycgo1ckyhl9zs43h1dd > > [4]: https://github.com/bazelbuild/intellij > > [5]: > > > https://github.com/confluentinc/kafka-bazel/blob/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/dependencies/dependencies.bzl#L24 > > [6]: > > > https://github.com/confluentinc/kafka-bazel/tree/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/tools-bazel > > [7]: > > > https://github.com/confluentinc/kafka-bazel/tree/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/kafka-overlay/dependencies > > [8]: > > > https://github.com/confluentinc/kafka-bazel/blob/a165c512c5d43256564b53ee0be2fbfe7388d5f9/kafka-bazel/WORKSPACE.bazel > > [9]: https://github.com/bazelbuild/starlark > > [10]: https://fzakaria.com/2024/08/29/bazel-overlay-pattern.html > > [11]: https://github.com/llvm/llvm-project/tree/main/utils/bazel > > > > > On 22 Nov 2024, at 03:39, Chia-Ping Tsai <chia7...@apache.org> wrote: > > > > > > hi Vince > > > > > > Thanks for introducing this interesting KIP! I'm not an expert in > Bazel, > > so please be patient with my potentially odd questions: > > > > > > CP0: Can Bazel help simplify build.gradle (currently around 3826 > lines)? > > > > > > CP1: Gradle commands are part of the public API, so in which Kafka > > version should we announce the deprecation of Gradle commands? > > > > > > CP2: The example (https://github.com/confluentinc/kafka-bazel) seems > to > > modify the folder structure of the entire Kafka project. Is this change > > required? My concern is that such a major change should be done in a > major > > release. If restructuring is a requirement for Bazel, does that mean we > > have to accept it after this KIP pass? > > > > > > Best, > > > Chia-Ping > > > > > > On 2024/11/21 21:16:50 Vince Rose wrote: > > >> Hey devs, > > >> > > >> I want to start a discussion about introducing Bazel as a build system > > for > > >> Apache Kafka. > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1115%3A+Bazel+Builds > > >> > > >> -Vince > > >> > > > > >
