This is an automated email from the ASF dual-hosted git repository.
jmclean pushed a commit to branch justinmclean-patch-4
in repository https://gitbox.apache.org/repos/asf/gravitino.git
commit 63d70f02b64de23af36e5b2615234ff0468ebaa0
Author: Justin Mclean <jus...@classsoftware.com>
AuthorDate: Mon Nov 18 09:22:30 2024 +1100
Improve how-to-build.md
---
docs/how-to-build.md | 217 +++++++++++++++++++++++----------------------------
1 file changed, 97 insertions(+), 120 deletions(-)
diff --git a/docs/how-to-build.md b/docs/how-to-build.md
index f929a840c..946ad3324 100644
--- a/docs/how-to-build.md
+++ b/docs/how-to-build.md
@@ -18,159 +18,136 @@ license: "This software is licensed under the Apache
License version 2."
:::info Please read the following notes before trying to build Gravitino.
-+ Gravitino requires a minimum of JDK8, and supports up to JDK17 to run
Gradle, so you need to install a JDK, version 8 to 17, to launch the build
environment.
-+ Gravitino itself supports using JDK8, 11, or 17 to build. The Gravitino
Trino connector uses
- JDK17 to build (to avoid vendor-related issues on some platforms, Gravitino
uses the specified Amazon Corretto OpenJDK 17 to build the Trino connector on
macOS).
- You don't have to preinstall the specified JDK environment, as Gradle
detects the JDK version needed and downloads it automatically.
-+ Gravitino uses the Gradle Java Toolchain to detect and manage JDK versions,
and it checks the
- installed JDK by running the `./gradlew javaToolchains` command. See [Gradle
Java
Toolchain](https://docs.gradle.org/current/userguide/toolchains.html#sec:java_toolchain).
-+ Gravitino excludes all Docker-related tests by default. To run
Docker-related tests, make sure you have installed
- Docker in your environment and either (1) set `skipDockerTests=false` in the
`gradle.properties` file (or
- use `-PskipDockerTests=false` in the command) or (2) `export
SKIP_DOCKER_TESTS=false` in shell. Otherwise, all tests requiring Docker will
be skipped.
-+ macOS uses `docker-connector` to make the Gravitino Trino connector work
with Docker
- for macOS. See
[docker-connector](https://github.com/wenjunxiao/mac-docker-connector),
`$GRAVITINO_HOME/dev/docker/tools/mac-docker-connector.sh`, and
- `$GRAVITINO_HOME/dev/docker/tools/README.md` for more details.
-+ You can use OrbStack as a replacement for Docker for macOS. See
- [OrbStack](https://orbstack.dev/). With OrbStack, you can run Gravitino
integration tests
- without needing to install `docker-connector`.
++ Gravitino requires a minimum of JDK8 and supports up to JDK17 to run Gradle,
so you need to install a JDK, version 8 to 17, to launch the build environment.
++ Gravitino itself supports using JDK 8, 11, or 17 to build. The Gravitino
Trino connector uses JDK17 to build (to avoid vendor-related issues on some
platforms, Gravitino uses the specified Amazon Corretto OpenJDK 17 to build the
Trino connector on macOS).
+ You don't have to preinstall the specified JDK environment, as Gradle detects
the JDK version needed and downloads it automatically.
++ Gravitino uses the Gradle Java Toolchain to detect and manage JDK versions,
and it checks the installed JDK by running the `./gradlew javaToolchains`
command. See [Gradle Java
Toolchain](https://docs.gradle.org/current/userguide/toolchains.html#sec:java_toolchain).
++ Gravitino excludes all Docker-related tests by default. To run
Docker-related tests, make sure you have installed Docker in your environment
and either (1) set `skipDockerTests=false` in the `gradle.properties` file (or
use `-PskipDockerTests=false` in the command) or (2) `export
SKIP_DOCKER_TESTS=false` in the shell. Otherwise, all tests requiring Docker
will be skipped.
++ macOS uses `docker-connector` to make the Gravitino Trino connector work
with Docker for macOS. See
[docker-connector](https://github.com/wenjunxiao/mac-docker-connector),
`$GRAVITINO_HOME/dev/docker/tools/mac-docker-connector.sh`, and
`$GRAVITINO_HOME/dev/docker/tools/README.md` for more details.
++ You can use OrbStack as a replacement for Docker for macOS. See
[OrbStack](https://orbstack.dev/). With OrbStack, you can run Gravitino
integration tests without needing to install `docker-connector`.
:::
## Quick start
1. Clone the Gravitino project.
- If you want to contribute to this open-source project, please fork the
project on GitHub first. After forking, clone the forked project to your local
environment, make your changes, and submit a pull request (PR).
+ If you want to contribute to this open-source project, please fork the
project on GitHub first. After forking, clone the forked project to your local
environment, make your changes, and submit a pull request (PR).
- ```shell
- git clone g...@github.com:apache/gravitino.git
- ```
+ ```shell
+ git clone g...@github.com:apache/gravitino.git
+ ```
2. Build the Gravitino project. Running this for the first time can take 15
minutes or more.
- ```shell
- cd gravitino
- ./gradlew build
- ```
+ ```shell
+ cd gravitino
+ ./gradlew build
+ ```
- The default specified JDK version is 8, but if you want to use JDK 11 or
17 to build,
- modify the property `jdkVersion` to 11 or 17 in the `gradle.properties`
file, or specify the version
- with `-P`, like:
+ The default specified JDK version is 8, but if you want to use JDK 11 or 17
to build,
+ modify the property `jdkVersion` to 11 or 17 in the `gradle.properties`
file, or specify the version
+ with `-P`, like:
- ```shell
- ./gradlew build -PjdkVersion=11
- ```
+ ```shell
+ ./gradlew build -PjdkVersion=11
+ ```
- Or:
+ Or:
- ```shell
- ./gradlew build -PjdkVersion=17
- ```
+ ```shell
+ ./gradlew build -PjdkVersion=17
+ ```
- The `./gradlew build` command builds all the Gravitino components,
including Gravitino
- server, Java and Python clients, Trino and Spark connectors, and more.
+ The `./gradlew build` command builds all the Gravitino components, including
the Gravitino server, Java and Python clients, Trino and Spark connectors, and
more.
- For the Python client, the `./gradlew build` command builds the Python
client with Python 3.8
- by default. If you want to use Python 3.9, 3.10 or 3.11 to build, please
modify the property
- `pythonVersion` to 3.9, 3.10 or 3.11 in the `gradle.properties` file, or
specify the version
- with `-P`, like:
+ For the Python client, the `./gradlew build` command builds the Python
client with Python 3.8 by default. If you want to use Python 3.9, 3.10, or 3.11
to build, please modify the property `pythonVersion` to 3.9, 3.10, or 3.11 in
the `gradle.properties` file, or specify the version with `-P` like:
- ```shell
- ./gradlew build -PpythonVersion=3.9
- ```
+ ```shell
+ ./gradlew build -PpythonVersion=3.9
+ ```
- Or:
+ Or:
- ```shell
- ./gradlew build -PpythonVersion=3.10
- ```
+ ```shell
+ ./gradlew build -PpythonVersion=3.10
+ ```
- Or:
+ Or:
- ```shell
- ./gradlew build -PpythonVersion=3.11
- ```
-
- If you want to build some modules alone, like Spark connector, you can use
Gradle build task
- with specific module name, like:
-
- ```shell
- ./gradlew spark-connector:spark-runtime-3.4:build -PscalaVersion=2.12
- ```
-
- This creates
`gravitino-spark-connector-runtime-{sparkVersion}_{scalaVersion}-{version}.jar`
- under the `spark-connector/v3.4/spark-runtime/build/libs` directory. You
could replace `3.4` with
- `3.3` or `3.5` to specify different Spark versions, and replace `2.12` with
`2.13` for different Scala
- versions. The default Scala version is `2.12` if not specifying
`-PscalaVersion`.
-
- :::info
- Gravitino Spark connector doesn't support Scala 2.13 for Spark 3.3.
- :::
-
- :::note
- The first time you build the project, downloading the dependencies may take
a while. You can add
- `-x test` to skip the tests using `./gradlew build -x test`.
-
- The built Gravitino libraries are Java 8 compatible and verified under Java
8, 11, and 17
- environments. You can use Java 8, 11, or 17 runtimes to run the Gravitino
server, no matter which
- JDK version you used to build the project.
-
- The built jars are under the `build/libs` directory of the modules. You can
publish them to your
- Maven repository to use them in your project.
- :::
+ ```shell
+ ./gradlew build -PpythonVersion=3.11
+ ```
-3. Get the Gravitino server binary package.
+ If you want to build a module on its own, like the Spark connector, you can
use Gradle to build a module with a specific name, like so:
- ```shell
- ./gradlew compileDistribution
- ```
+ ```shell
+ ./gradlew spark-connector:spark-runtime-3.4:build -PscalaVersion=2.12
+ ```
- The `compileDistribution` command creates a `distribution` directory in the
Gravitino root directory.
+ This creates
`gravitino-spark-connector-runtime-{sparkVersion}_{scalaVersion}-{version}.jar`
under the `spark-connector/v3.4/spark-runtime/build/libs` directory. You could
replace `3.4` with `3.3` or `3.5` to specify different Spark versions and
replace `2.12` with `2.13` for different Scala versions. The default Scala
version is `2.12` if `-PscalaVersion` is not specified.
- The directory structure of the `distribution` directory is as follows:
+ :::info
+ Gravitino Spark connector doesn't support Scala 2.13 for Spark 3.3.
+ :::
- :::note
- The `./gradlew clean` command deletes the `distribution` directory.
- :::
+ :::note
+ The first time you build the project, downloading the dependencies may take
a while.
+
+ You can add `-x test` to skip the tests using `./gradlew build -x test`.
+
+ The built Gravitino libraries are Java 8 compatible and verified under the
Java 8, 11, and 17 environments. You can use Java 8, 11, or 17 runtimes to run
the Gravitino server, no matter which JDK version was used to build the project.
+
+ The built jars are under the modules `build/libs` directory. You can publish
them in your Maven repository for use in your project.
+ :::
+
+3. Get the Gravitino server binary package.
+
+ ```shell
+ ./gradlew compileDistribution
+ ```
+
+ The `compileDistribution` command creates a `distribution` directory in the
Gravitino root directory.
+
+ :::note
+ The `./gradlew clean` command deletes the `distribution` directory.
+ :::
4. Assemble the Gravitino server distribution package.
- ```shell
- ./gradlew assembleDistribution
- ```
+ ```shell
+ ./gradlew assembleDistribution
+ ```
- The `assembleDistribution` command creates `gravitino-{version}-bin.tar.gz`
and `gravitino-{version}-bin.tar.gz.sha256` under the `distribution` directory.
+ The `assembleDistribution` command creates `gravitino-{version}-bin.tar.gz`
and `gravitino-{version}-bin.tar.gz.sha256` under the `distribution` directory.
- You can deploy them to your production environment.
+ You can deploy these to your production environment.
- :::note
- The `gravitino-{version}-bin.tar.gz` file is the Gravitino **server**
distribution package, and the
- `gravitino-{version}-bin.tar.gz.sha256` file is the sha256 checksum file
for the Gravitino
- server distribution package.
- :::
+ :::note
+ The `gravitino-{version}-bin.tar.gz` file is the Gravitino **server**
distribution package, and the `gravitino-{version}-bin.tar.gz.sha256` file is
the sha256 checksum file for the Gravitino server distribution package.
+ :::
5. Assemble the Gravitino Trino connector package
- ```shell
- ./gradlew assembleTrinoConnector
- ```
+ ```shell
+ ./gradlew assembleTrinoConnector
+ ```
- or
+ or
- ```shell
- ./gradlew assembleDistribution
- ```
+ ```shell
+ ./gradlew assembleDistribution
+ ```
- This creates `gravitino-trino-connector-{version}.tar.gz` and
- `gravitino-trino-connector-{version}.tar.gz.sha256` under the
`distribution` directory. You
- can uncompress and deploy it to Trino to use the Gravitino Trino connector.
+ This creates `gravitino-trino-connector-{version}.tar.gz` and
+ `gravitino-trino-connector-{version}.tar.gz.sha256` under the `distribution`
directory. You can uncompress and deploy it to Trino to use the Gravitino Trino
connector.
6. Assemble the Gravitino Iceberg REST server package
- ```shell
- ./gradlew assembleIcebergRESTServer
- ```
+ ```shell
+ ./gradlew assembleIcebergRESTServer
+ ```
- This creates `gravitino-iceberg-rest-server-{version}.tar.gz` and
`gravitino-iceberg-rest-server-{version}.tar.gz.sha256` under the
`distribution` directory. You can uncompress and deploy it to use the Gravitino
Iceberg REST server.
+ This creates `gravitino-iceberg-rest-server-{version}.tar.gz` and
`gravitino-iceberg-rest-server-{version}.tar.gz.sha256` under the
`distribution` directory. You can uncompress and deploy it to use the Gravitino
Iceberg REST server.
## How to build Apache Gravitino on Windows (Using WSL)
@@ -180,7 +157,7 @@ license: "This software is licensed under the Apache
License version 2."
Refer to this guide for installation: [WSL Installation
Guide](https://learn.microsoft.com/en-us/windows/wsl/install)
-*Note: Ubuntu 22.04 can successfully run Gravitino*
+*Note: Gravitino can run successfully on Ubuntu 22.04*
This step involves setting up your Windows machine's Windows Subsystem for
Linux (WSL). WSL allows you to run a Linux distribution alongside Windows,
providing a Linux-like environment for development.
@@ -201,22 +178,22 @@ Updating the package list ensures you have the latest
information on the newest
1. Edit your `~/.bashrc` file using any editor. Here, `vim` is used:
- ```shell
- vim ~/.bashrc
- ```
+ ```shell
+ vim ~/.bashrc
+ ```
2. Add the following lines at the end of the file. Replace
`/usr/lib/jvm/java-11-openjdk-amd64` with your actual Java installation path:
- ```sh
- export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
- export PATH=$PATH:$JAVA_HOME/bin
- ```
+ ```sh
+ export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
+ export PATH=$PATH:$JAVA_HOME/bin
+ ```
3. Save and quit in vim using `:wq`.
4. Run `source ~/.bashrc` to update your shell session's environment variables.
- Editing the `~/.bashrc` file allows you to set environment variables
available in every terminal session. Setting `JAVA_HOME` and updating `PATH`
ensures that your system uses the correct Java version for development.
+ Editing the `~/.bashrc` file allows you to set environment variables
available in every terminal session. Setting `JAVA_HOME` and updating `PATH`
ensures that your system uses the correct Java version for development.
### Install Docker
