tengqm commented on code in PR #6659: URL: https://github.com/apache/gravitino/pull/6659#discussion_r1990510440
########## docs/how-to-build.md: ########## @@ -4,244 +4,232 @@ slug: /how-to-build license: "This software is licensed under the Apache License version 2." --- -- [Prerequisites](#prerequisites) -- [Quick start](#quick-start) -- [How to Build Apache Gravitino on Windows (Using WSL)](#how-to-build-apache-gravitino-on-windows-using-wsl) - ## Prerequisites -+ Linux or macOS operating system -+ Git -+ A Java Development Kit, version 8 to 17, installed in your environment to launch Gradle -+ Python 3.8, 3.9, 3.10, or 3.11 to build the Gravitino Python client -+ Optionally, Docker to run integration tests - -:::info Please read the following notes before trying to build Gravitino. - -+ Gravitino requires a minimum of JDK8 and supports up to JDK17 to run Gradle, so you need to install a JDK, version 8 to 17, to launch the build environment. -+ Gravitino itself supports using JDK 8, 11, or 17 to build. The Gravitino Trino connector uses JDK17 to build (to avoid vendor-related issues on some platforms, Gravitino uses the specified Amazon Corretto OpenJDK 17 to build the Trino connector on macOS). - You don't have to preinstall the specified JDK environment, as Gradle detects the JDK version needed and downloads it automatically. -+ Gravitino uses the Gradle Java Toolchain to detect and manage JDK versions, and it checks the installed JDK by running the `./gradlew javaToolchains` command. See [Gradle Java Toolchain](https://docs.gradle.org/current/userguide/toolchains.html#sec:java_toolchain). -+ Gravitino excludes all Docker-related tests by default. To run Docker-related tests, make sure you have installed Docker in your environment and either (1) set `skipDockerTests=false` in the `gradle.properties` file (or use `-PskipDockerTests=false` in the command) or (2) `export SKIP_DOCKER_TESTS=false` in the shell. Otherwise, all tests requiring Docker will be skipped. -+ macOS uses `docker-connector` to make the Gravitino Trino connector work with Docker for macOS. See [docker-connector](https://github.com/wenjunxiao/mac-docker-connector), `$GRAVITINO_HOME/dev/docker/tools/mac-docker-connector.sh`, and `$GRAVITINO_HOME/dev/docker/tools/README.md` for more details. -+ You can use OrbStack as a replacement for Docker for macOS. See [OrbStack](https://orbstack.dev/). With OrbStack, you can run Gravitino integration tests without needing to install `docker-connector`. -+ Depending on how you deploy Gravitino, other software used in conjunction with Gravitino may contain known security vulnerabilities. -::: - -## Quick start +- Linux, macOS or Windows operating system +- Git +- JDK 8 (default), 11 or 17 for running Gradle, the build environment for Gravitino. +- Python 3.8 (default), 3.9, 3.10, or 3.11, for building the Gravitino Python client +- Docker (optional), for running the integration tests -1. Clone the Gravitino project. - - If you want to contribute to this open-source project, please fork the project on GitHub first. After forking, clone the forked project to your local environment, make your changes, and submit a pull request (PR). +:::note +Depending on how you deploy Gravitino, there may be contain known security vulnerabilities +in other software used in conjunction with Gravitino. +::: - ```shell - git clone g...@github.com:apache/gravitino.git - ``` +### Install WSL on Windows -2. Build the Gravitino project. Running this for the first time can take 15 minutes or more. +If you are building Gravitino on a Linux or macOS machine, you can proceed to the [next section](#install-jdk). +To build Apache Gravitino on Windows, you need to install the Windows Subsystem for Linux (WSL) +from Microsoft which provides you with a Linux-like environment for development. +To install WSL, refer to the [WSL Installation Guide](https://learn.microsoft.com/en-us/windows/wsl/install). +The following steps assume that you have installed Unbuntu, which is the default distribution for WSL. - ```shell - cd gravitino - ./gradlew build - ``` +*Note: Gravitino can run successfully on Ubuntu 22.04.* - The default specified JDK version is 8, but if you want to use JDK 11 or 17 to build, modify the property `jdkVersion` to 11 or 17 in the `gradle.properties` file, or specify the version with `-P`, like: +Make sure you have fresh information about the latest versions of Ubuntu packages and their dependencies: - ```shell - ./gradlew build -PjdkVersion=11 - ``` +```shell +sudo apt update +``` - Or: +Install the prerequisite packages required by later software installation: - ```shell - ./gradlew build -PjdkVersion=17 - ``` +```shell +sudo apt install apt-transport-https ca-certificates curl software-properties-common +``` - The `./gradlew build` command builds all the Gravitino components, including the Gravitino server, Java and Python clients, Trino and Spark connectors, and more. +### Install JDK - For the Python client, the `./gradlew build` command builds the Python client with Python 3.8 by default. If you want to use Python 3.9, 3.10, or 3.11 to build, please modify the property `pythonVersion` to 3.9, 3.10, or 3.11 in the `gradle.properties` file, or specify the version with `-P` like: +You don't have to pre-install the JDK environment. +The Gradle Java toolchain can detect and install JDK automatically by running the `./gradlew javaToolchains` command. +For more details, see [Gradle Java Toolchain](https://docs.gradle.org/current/userguide/toolchains.html#sec:java_toolchain). - ```shell - ./gradlew build -PpythonVersion=3.9 - ``` +If you want to install JDK manually, Java SDK 17 is recommended, and JDK 8 or JDK 11 also works. +After installing the JDK, you may want to set up your Shell environment to ensure that the correct JDK version will be used. - Or: +1. Edit the `~/.bashrc` file by adding the following lines at the end of the file. - ```shell - ./gradlew build -PpythonVersion=3.10 + ```sh + export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64" + export PATH=$PATH:$JAVA_HOME/bin ``` - Or: + Replace `/usr/lib/jvm/java-11-openjdk-amd64` with your actual Java installation path. - ```shell - ./gradlew build -PpythonVersion=3.11 - ``` +1. Run `source ~/.bashrc` to update your Shell environment variables. - If you want to build a module on its own, like the Spark connector, you can use Gradle to build a module with a specific name, like so: +### Install Python - ```shell - ./gradlew spark-connector:spark-runtime-3.4:build -PscalaVersion=2.12 - ``` +Python 3.8 is the default version for building the Gravitino Python client packages.. +Python 3.9, 3.10, or 3.11 also works. - This creates `gravitino-spark-connector-runtime-{sparkVersion}_{scalaVersion}-{version}.jar` under the `spark-connector/v3.4/spark-runtime/build/libs` directory. You could replace `3.4` with `3.3` or `3.5` to specify different Spark versions and replace `2.12` with `2.13` for different Scala versions. The default Scala version is `2.12` if `-PscalaVersion` is not specified. +Add a repository for the latest Python versions and install Python 3.8: - :::info - Gravitino Spark connector doesn't support Scala 2.13 for Spark 3.3. - ::: +```shell +sudo add-apt-repository ppa:deadsnakes/ppa +sudo apt update +sudo apt install python3.8 +``` - :::note - The first time you build the project, downloading the dependencies may take a while. - - You can add `-x test` to skip the tests using `./gradlew build -x test`. +### Install Docker - The built Gravitino libraries are Java 8 compatible and verified under the Java 8, 11, and 17 environments. You can use Java 8, 11, or 17 runtimes to run the Gravitino server, no matter which JDK version was used to build the project. +If you don't want to run tests, you can proceed to the [next step](#build-gravitino). - The built jars are under the modules `build/libs` directory. You can publish them in your Maven repository for use in your project. - ::: +:::info +- Gravitino skips all Docker-related tests by default. -3. Get the Gravitino server binary package. + To run Docker-related tests, make sure you have installed Docker, and + (1) set `skipDockerTests=false` in the `gradle.properties` file, or + (2) use `-PskipDockerTests=false` in the command line, or + (3) `export SKIP_DOCKER_TESTS=false` in the Shell. - ```shell - ./gradlew compileDistribution - ``` +<!--TODO(Qiming): move the following two items elsewhere--> +- macOS uses [docker-connector](https://github.com/wenjunxiao/mac-docker-connector) + to make the Gravitino Trino connector work. Refer to + `$GRAVITINO_HOME/dev/docker/tools/mac-docker-connector.sh`, and + `$GRAVITINO_HOME/dev/docker/tools/README.md` for more details. + The Gravitino Trino connector uses JDK17 to build. + For example, to avoid vendor-related issues on macOS, + Gravitino uses the Amazon Corretto OpenJDK 17 to build the Trino connector. - The `compileDistribution` command creates a `distribution` directory in the Gravitino root directory. +- You can use [OrbStack](https://orbstack.dev) as an alternative for Docker for macOS. + With OrbStack, you can run Gravitino integration tests without `docker-connector`. +::: - :::note - The `./gradlew clean` command deletes the `distribution` directory. - ::: -4. Assemble the Gravitino server distribution package. +1. Add the apt repository for Docker-CE: ```shell - ./gradlew assembleDistribution + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - + sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" ``` - The `assembleDistribution` command creates `gravitino-{version}-bin.tar.gz` and `gravitino-{version}-bin.tar.gz.sha256` under the `distribution` directory. - - You can deploy these to your production environment. - - :::note - The `gravitino-{version}-bin.tar.gz` file is the Gravitino **server** distribution package, and the `gravitino-{version}-bin.tar.gz.sha256` file is the sha256 checksum file for the Gravitino server distribution package. - ::: - -5. Assemble the Gravitino Trino connector package +1. Install the `docker-ce` package and start the Docker daemon: - ```shell - ./gradlew assembleTrinoConnector + ```shell + sudo apt update + sudo apt install docker-ce + sudo service docker start ``` - or +1. Verify the installation by running `hello-world`: ```shell - ./gradlew assembleDistribution + sudo docker run hello-world ``` - This creates `gravitino-trino-connector-{version}.tar.gz` and - `gravitino-trino-connector-{version}.tar.gz.sha256` under the `distribution` directory. You can uncompress and deploy it to Trino to use the Gravitino Trino connector. - -6. Assemble the Gravitino Iceberg REST server package +1. Add yourself to the `docker` group so that you can run Docker commands without `sudo` in the future. - ```shell - ./gradlew assembleIcebergRESTServer + ```shell + sudo usermod -aG docker $USER ``` - This creates `gravitino-iceberg-rest-server-{version}.tar.gz` and `gravitino-iceberg-rest-server-{version}.tar.gz.sha256` under the `distribution` directory. You can uncompress and deploy it to use the Gravitino Iceberg REST server. - -## How to build Apache Gravitino on Windows (Using WSL) +## Build Gravitino -### Download WSL (Ubuntu) - -**On Windows:** - -Refer to this guide for installation: [WSL Installation Guide](https://learn.microsoft.com/en-us/windows/wsl/install) - -*Note: Gravitino can run successfully on Ubuntu 22.04* - -This step involves setting up your Windows machine's Windows Subsystem for Linux (WSL). WSL allows you to run a Linux distribution alongside Windows, providing a Linux-like environment for development. - -### Update package list and install necessary packages - -**On Ubuntu (WSL):** - -```shell -sudo apt update -sudo apt install apt-transport-https ca-certificates curl software-properties-common -``` +1. Clone the Gravitino project. -Updating the package list ensures you have the latest information on the newest versions of packages and dependencies. Installing the necessary packages lets your system download and manage additional software securely. + ```shell + git clone g...@github.com:apache/gravitino.git + ``` -### Download and setup Java SDK 17 (11 or 8 also works) +1. Build the Gravitino project. Running this for the first time can take 15 minutes or more. -**On Ubuntu (WSL):** + ```shell + cd gravitino + ./gradlew build + ``` -1. Edit your `~/.bashrc` file using any editor. Here, `vim` is used: + You can customize the property `jdkVersion` in the `gradle.properties` file to use JDK 11 or JDK 17. + Alternatively, you can specify the version using the `-P` Gradle flag. e.g. ```shell - vim ~/.bashrc + ./gradlew build -PjdkVersion=11 ``` -2. Add the following lines at the end of the file. Replace `/usr/lib/jvm/java-11-openjdk-amd64` with your actual Java installation path: - - ```sh - export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64" - export PATH=$PATH:$JAVA_HOME/bin + The `./gradlew build` command builds all the Gravitino components, + including the Gravitino server, Java and Python clients, Trino and Spark connectors etc. + + You can customize the `pythonVersion` property in the `gradle.properties` file + to use Python 3.9, 3.10, or 3.11. Alternatively, you can set it on the command line + using the `-P` Gradle flag. e.g. + + ```shell + ./gradlew build -PpythonVersion=3.9 ``` -3. Save and quit in vim using `:wq`. + :::note + The Gravitino libraries built are Java 8 compatible and verified under Java 8, 11, and 17. + You can run the Gravitino server using JRE 8, 11, or 17, + regardless which JDK version was used to build the project. Review Comment: I can fix this. But, again, this issue is **not** caused by line breaking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
