Hi Salva, unfortunately, I have no experience with Bazel. Just by looking at the code you shared I cannot come up with an answer either. Have you checked out the ML thread in [1]? It provides two other examples where users used Bazel in the context of Flink. This might give you some hints on where to look.
Sorry for not being more helpful. Best, Matthias [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-anyone-have-an-example-of-Bazel-working-with-Flink-td35898.html On Fri, Apr 30, 2021 at 11:57 AM Salva Alcántara <salcantara...@gmail.com> wrote: > I am trying to setup a simple flink application from scratch using Bazel. > I've bootstrapped the project by running > > ``` > sbt new tillrohrmann/flink-project.g8 > ``` > > and after that I have added some files in order for Bazel to take control > of > the building (i.e., migrate from sbt). This is how the `WORKSPACE` looks > like > > ``` > # WORKSPACE > load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") > > skylib_version = "1.0.3" > http_archive( > name = "bazel_skylib", > sha256 = > "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c", > type = "tar.gz", > url = > " > https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib-{}.tar.gz > ".format(skylib_version, > skylib_version), > ) > > rules_scala_version = "5df8033f752be64fbe2cedfd1bdbad56e2033b15" > > http_archive( > name = "io_bazel_rules_scala", > sha256 = > "b7fa29db72408a972e6b6685d1bc17465b3108b620cb56d9b1700cf6f70f624a", > strip_prefix = "rules_scala-%s" % rules_scala_version, > type = "zip", > url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip"; % > rules_scala_version, > ) > > # Stores Scala version and other configuration > # 2.12 is a default version, other versions can be use by passing them > explicitly: > load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config") > scala_config(scala_version = "2.12.11") > > load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories") > scala_repositories() > > load("@io_bazel_rules_scala//scala:toolchains.bzl", > "scala_register_toolchains") > scala_register_toolchains() > > load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", > "scala_binary", "scala_test") > > # optional: setup ScalaTest toolchain and dependencies > load("@io_bazel_rules_scala//testing:scalatest.bzl", > "scalatest_repositories", "scalatest_toolchain") > scalatest_repositories() > scalatest_toolchain() > > load("//vendor:workspace.bzl", "maven_dependencies") > maven_dependencies() > > load("//vendor:target_file.bzl", "build_external_workspace") > build_external_workspace(name = "vendor") > ``` > > and this is the `BUILD` file > > ```bazel > package(default_visibility = ["//visibility:public"]) > > load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", > "scala_test") > > scala_library( > name = "job", > srcs = glob(["src/main/scala/**/*.scala"]), > deps = [ > "@vendor//vendor/org/apache/flink:flink_clients", > "@vendor//vendor/org/apache/flink:flink_scala", > "@vendor//vendor/org/apache/flink:flink_streaming_scala", > ] > ) > ``` > > I'm using [bazel-deps](https://github.com/johnynek/bazel-deps) for > vendoring > the dependencies (put in the `vendor` folder). I have this on my > `dependencies.yaml` file: > > ```yaml > options: > buildHeader: [ > "load(\"@io_bazel_rules_scala//scala:scala_import.bzl\", > \"scala_import\")", > "load(\"@io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", > \"scala_binary\", \"scala_test\")", > ] > languages: [ "java", "scala:2.12.11" ] > resolverType: "coursier" > thirdPartyDirectory: "vendor" > resolvers: > - id: "mavencentral" > type: "default" > url: https://repo.maven.apache.org/maven2/ > strictVisibility: true > transitivity: runtime_deps > versionConflictPolicy: highest > > dependencies: > org.apache.flink: > flink: > lang: scala > version: "1.11.2" > modules: [clients, scala, streaming-scala] # provided > flink-connector-kafka: > lang: java > version: "0.10.2" > flink-test-utils: > lang: java > version: "0.10.2" > ``` > > For downloading the dependencies, I'm running > > ``` > bazel run //:parse generate -- --repo-root ~/Projects/bazel-flink-scala > --sha-file vendor/workspace.bzl --target-file vendor/target_file.bzl --deps > dependencies.yaml > ``` > > Which runs just fine, but then when I try to build the project > > ``` > bazel build //:job > ``` > > I'm getting this error > > ``` > Starting local Bazel server and connecting to it... > ERROR: Traceback (most recent call last): > File > "/Users/salvalcantara/Projects/me/bazel-flink-scala/WORKSPACE", line > 44, column 25, in <toplevel> > build_external_workspace(name = "vendor") > File > > "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", > line 258, column 91, in build_external_workspace > return build_external_workspace_from_opts(name = name, > target_configs = > list_target_data(), separator = list_target_data_separator(), build_header > = > build_header()) > File > > "/Users/salvalcantara/Projects/me/bazel-flink-scala/vendor/target_file.bzl", > line 251, column 40, in list_target_data > "vendor/org/apache/flink:flink_clients": > > ["lang||||||scala:2.12.11","name||||||//vendor/org/apache/flink:flink_clients","visibility||||||//visibility:public","kind||||||import","deps|||L|||","jars|||L|||//external:jar/org/apache/flink/flink_clients_2_12","sources|||L|||","exports|||L|||","runtimeDeps|||L|||//vendor/commons_cli:commons_cli|||//vendor/org/slf4j:slf4j_api|||//vendor/org/apache/flink:force_shading|||//vendor/com/google/code/findbugs:jsr305|||//vendor/org/apache/flink:flink_streaming_java_2_12|||//vendor/org/apache/flink:flink_core|||//vendor/org/apache/flink:flink_java|||//vendor/org/apache/flink:flink_runtime_2_12|||//vendor/org/apache/flink:flink_optimizer_2_12","processorClasses|||L|||","generatesApi|||B|||false","licenses|||L|||","generateNeverlink|||B|||false"], > Error: dictionary expression has duplicate key: > "vendor/org/apache/flink:flink_clients" > ERROR: error loading package 'external': Package 'external' contains errors > INFO: Elapsed time: 3.644s > INFO: 0 processes. > FAILED: Build did NOT complete successfully (0 packages loaded) > ``` > > Why is that? Anyone can help? It would be great having detailed > instructions > and project templates for Flink/Scala applications using Bazel. I've put > everything together in the following repo: > https://github.com/salvalcantara/bazel-flink-scala, feel free to send a PR > or whatever. > > PS: Also posted in SO: > > https://stackoverflow.com/questions/67331792/setup-of-scala-flink-project-using-bazel > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
