dnskr commented on code in PR #7541: URL: https://github.com/apache/kyuubi/pull/7541#discussion_r3522119416
########## docs/quick_start/quick_start.md: ########## @@ -0,0 +1,282 @@ +<!-- +- Licensed to the Apache Software Foundation (ASF) under one or more +- contributor license agreements. See the NOTICE file distributed with +- this work for additional information regarding copyright ownership. +- The ASF licenses this file to You under the Apache License, Version 2.0 +- (the "License"); you may not use this file except in compliance with +- the License. You may obtain a copy of the License at +- +- http://www.apache.org/licenses/LICENSE-2.0 +- +- Unless required by applicable law or agreed to in writing, software +- distributed under the License is distributed on an "AS IS" BASIS, +- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +- See the License for the specific language governing permissions and +- limitations under the License. +--> + +# Getting Started + +```{note} +This page covers how to start with kyuubi quickly on your laptop in about 3~5 minutes. +``` + +## Requirements + +For quick start deployment, we need to prepare the following stuffs: + +- A **client** that connects and submits queries to the server. Here, we use the kyuubi beeline for demonstration. +- A **server** that serves clients and manages engines. +- An **engine** that is used to instantiate query execution environments. Here we use Spark for demonstration. + +These essential components are JVM-based applications. +So, the JRE needs to be pre-installed and the `JAVA_HOME` is correctly set to each component. + +| Component | Role | Version | Remarks | +|:--------------|:---------------------------------|:------------------------------|:--------------------------------------------------------------------------------------------------------| +| **Java** | JRE | 8, 11, 17 | Officially released against JDK8 | +| **Kyuubi** | Gateway<br>Engine lib<br>Beeline | {{ release }} | - Kyuubi Server<br>- Kyuubi Engine<br>- Kyuubi Beeline | +| **Spark** | Engine | 3.3 to 3.5, 4.0 to 4.1 | A Spark distribution | +| **Flink** | Engine | 1.17 to 1.20 | A Flink distribution | +| **Trino** | Engine | N/A | A Trino cluster allows to access via trino-client v411 | +| **Doris** | Engine | N/A | A Doris cluster | +| **Hive** | Engine<br>Metastore | - 2.1-cdh6, 2.3, 3.1<br>- N/A | - A Hive distribution<br>- An optional and external metadata store, whose version is decided by engines | +| **Zookeeper** | HA | >=3.4.x | | +| **Disk** | Storage | N/A | N/A | + +The other internal or external parts listed in the above sheet can be used individually or all together. +For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. +And then, you can use Zookeeper to enable the load balancing for high availability. +The data could be stored in Hive, Apache Iceberg, or other DBMSs. + +In what follows, we will only use Kyuubi and Spark. + +## Installation + +```{note} +:class: dropdown, toggle + +This following instructions are based on binary releases. +If you start with source releases, please refer to the page for [building kyuubi](../contributing/code/distribution.md). +``` + +### Install Kyuubi + +The official releases, binary- and source-, are archived on the [download page](https://kyuubi.apache.org/releases.html). +Please download the most recent stable release to start. + +To install Kyuubi, you need to unpack the tarball. For example, + +```{parsed-literal} +$ tar zxf apache-kyuubi-{{ release }}-bin.tgz +``` Review Comment: Substitution doesn't work in standard markdown editors (viewers), see [quick_start.md](https://github.com/dnskr/kyuubi/blob/91931a9efd9d2d83425fbb58659469be4341797e/docs/quick_start/quick_start.md#install-kyuubi) <img width="1234" height="460" alt="Image" src="https://github.com/user-attachments/assets/89223e58-63f6-4cc9-8454-0530f6ae3ce0"; /> ########## docs/conf.py: ########## @@ -70,11 +70,18 @@ "sphinx.ext.mathjax", "myst_parser", "sphinx_copybutton", - "sphinx_markdown_tables", Review Comment: Need to be deleted to fix substitution in markdown tables. ########## docs/quick_start/quick_start.md: ########## @@ -0,0 +1,282 @@ +<!-- +- Licensed to the Apache Software Foundation (ASF) under one or more +- contributor license agreements. See the NOTICE file distributed with +- this work for additional information regarding copyright ownership. +- The ASF licenses this file to You under the Apache License, Version 2.0 +- (the "License"); you may not use this file except in compliance with +- the License. You may obtain a copy of the License at +- +- http://www.apache.org/licenses/LICENSE-2.0 +- +- Unless required by applicable law or agreed to in writing, software +- distributed under the License is distributed on an "AS IS" BASIS, +- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +- See the License for the specific language governing permissions and +- limitations under the License. +--> + +# Getting Started + +```{note} +This page covers how to start with kyuubi quickly on your laptop in about 3~5 minutes. +``` + +## Requirements + +For quick start deployment, we need to prepare the following stuffs: + +- A **client** that connects and submits queries to the server. Here, we use the kyuubi beeline for demonstration. +- A **server** that serves clients and manages engines. +- An **engine** that is used to instantiate query execution environments. Here we use Spark for demonstration. + +These essential components are JVM-based applications. +So, the JRE needs to be pre-installed and the `JAVA_HOME` is correctly set to each component. + +| Component | Role | Version | Remarks | +|:--------------|:---------------------------------|:------------------------------|:--------------------------------------------------------------------------------------------------------| +| **Java** | JRE | 8, 11, 17 | Officially released against JDK8 | +| **Kyuubi** | Gateway<br>Engine lib<br>Beeline | {{ release }} | - Kyuubi Server<br>- Kyuubi Engine<br>- Kyuubi Beeline | +| **Spark** | Engine | 3.3 to 3.5, 4.0 to 4.1 | A Spark distribution | +| **Flink** | Engine | 1.17 to 1.20 | A Flink distribution | +| **Trino** | Engine | N/A | A Trino cluster allows to access via trino-client v411 | +| **Doris** | Engine | N/A | A Doris cluster | +| **Hive** | Engine<br>Metastore | - 2.1-cdh6, 2.3, 3.1<br>- N/A | - A Hive distribution<br>- An optional and external metadata store, whose version is decided by engines | +| **Zookeeper** | HA | >=3.4.x | | +| **Disk** | Storage | N/A | N/A | + +The other internal or external parts listed in the above sheet can be used individually or all together. +For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. +And then, you can use Zookeeper to enable the load balancing for high availability. +The data could be stored in Hive, Apache Iceberg, or other DBMSs. + +In what follows, we will only use Kyuubi and Spark. + +## Installation + +```{note} +:class: dropdown, toggle + +This following instructions are based on binary releases. +If you start with source releases, please refer to the page for [building kyuubi](../contributing/code/distribution.md). +``` + +### Install Kyuubi + +The official releases, binary- and source-, are archived on the [download page](https://kyuubi.apache.org/releases.html). +Please download the most recent stable release to start. + +To install Kyuubi, you need to unpack the tarball. For example, + +```{parsed-literal} +$ tar zxf apache-kyuubi-{{ release }}-bin.tgz +``` + +```{code-block} +--- +class: toggle +--- +├── LICENSE +├── NOTICE +├── RELEASE +├── beeline-jars +├── bin +├── charts +│ └── kyuubi +├── conf +| ├── kyuubi-defaults.conf.template +│ ├── kyuubi-env.sh.template +│ └── log4j2.xml.template +├── db-scripts +│ ├── mysql +│ ├── postgresql +│ └── sqlite +├── docker +│ ├── Dockerfile +│ └── playground +├── externals +│ └── engines +├── jars +├── licenses +├── logs +├── pid +├── web-ui +└── work +``` + +From top to bottom are: + +- LICENSE: the APACHE [LICENSE](https://www.apache.org/licenses/LICENSE-2.0), VERSION 2.0 we claim to obey. +- RELEASE: the build information of this package. +- NOTICE: the notice made by Apache Kyuubi Community about its project and dependencies. +- bin: the entry of the Kyuubi server with `kyuubi` as the startup script. +- conf: all the defaults used by Kyuubi Server itself or creating a session with engines. +- externals + - engines: contains all kinds of SQL engines that we support +- licenses: a bunch of licenses included. +- jars: packages needed by the Kyuubi server. +- logs: where the logs of the Kyuubi server locates. +- pid: stores the PID file of the Kyuubi server instance. +- work: the root of the working directories of all the forked sub-processes, a.k.a. SQL engines. + +### Install Spark + +The official releases, binary- and source-, are archived on the [spark download page](https://spark.apache.org/downloads.html). +Please download the most recent stable release to start. + +:::{note} +:class: dropdown, toggle + +Currently, Kyuubi is compiled and pre-built against Spark 3 and Scala 2.12. +You will probably meet runtime exceptions if you use Spark 2 or Spark with unsupported Scala versions. +::: + +To install Spark, you need to unpack the tarball. For example, + +```shell +$ tar zxf spark-3.4.2-bin-hadoop3.tgz +``` + +### Configuration + +The `kyuubi-env.sh` file is used to set system environment variables to the kyuubi server process and engine processes it creates. + +The `kyuubi-defaults.conf` file is used to set system properties to the kyuubi server process and engine processes it creates. + +Each file has a template lays in `conf` directory for your information. +The following are examples of the parameters necessary for a quick start with Spark. + +- **JAVA_HOME** + +```shell +$ echo 'export JAVA_HOME=/path/to/java' >> conf/kyuubi-env.sh +``` + +- **SPARK_HOME** + +```shell +$ echo 'export SPARK_HOME=/path/to/spark' >> conf/kyuubi-env.sh +``` + +## Start Kyuubi + +```shell +$ bin/kyuubi start +``` + +If script above runs successfully, it will store the `PID` of the server instance +into `pid/kyuubi-<username>-org.apache.kyuubi.server.KyuubiServer.pid`. +And you are able to get the JDBC connection URL from the log file - +`logs/kyuubi-<username>-org.apache.kyuubi.server.KyuubiServer-<hostname>.out`. + +For example, + +```{code-block} +Starting and exposing JDBC connection at: jdbc:kyuubi://localhost:10009/ +``` + +If something goes wrong, you shall be able to find some clues in the log file too. + +:::{note} +:class: toggle + +Alternatively, it can run in the foreground, with the logs and other output +written to stdout/stderr. Both streams should be captured if using a +supervision system like `supervisord`. + +```shell +bin/kyuubi run +``` +::: Review Comment: Directive uses `:::` delimiters instead of ```` ``` ```` to render `shell` code correctly. However, in standard markdown editors (viewers) it looks as [text](https://github.com/dnskr/kyuubi/blob/91931a9efd9d2d83425fbb58659469be4341797e/docs/quick_start/quick_start.md#start-kyuubi). <img width="1238" height="638" alt="Image" src="https://github.com/user-attachments/assets/23b35ab7-6ede-470d-b001-4b2c5357f2c7"; /> ########## docs/quick_start/index.md: ########## @@ -0,0 +1,32 @@ +<!-- +- Licensed to the Apache Software Foundation (ASF) under one or more +- contributor license agreements. See the NOTICE file distributed with +- this work for additional information regarding copyright ownership. +- The ASF licenses this file to You under the Apache License, Version 2.0 +- (the "License"); you may not use this file except in compliance with +- the License. You may obtain a copy of the License at +- +- http://www.apache.org/licenses/LICENSE-2.0 +- +- Unless required by applicable law or agreed to in writing, software +- distributed under the License is distributed on an "AS IS" BASIS, +- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +- See the License for the specific language governing permissions and +- limitations under the License. +--> + +# Quick Start + +```{note} +In this section, you will learn how to setup and interact with kyuubi quickly. +``` + +```{toctree} +--- +maxdepth: 2 +--- Review Comment: Directive properties are rendered as text in standard markdown editors (viewers), see [index.md](https://github.com/dnskr/kyuubi/blob/91931a9efd9d2d83425fbb58659469be4341797e/docs/quick_start/index.md) <img width="1237" height="411" alt="Image" src="https://github.com/user-attachments/assets/fb03c709-2aa4-4a64-9932-18870c35a7c2"; /> ########## docs/quick_start/quick_start_with_jdbc.md: ########## @@ -0,0 +1,92 @@ +<!-- +- Licensed to the Apache Software Foundation (ASF) under one or more +- contributor license agreements. See the NOTICE file distributed with +- this work for additional information regarding copyright ownership. +- The ASF licenses this file to You under the Apache License, Version 2.0 +- (the "License"); you may not use this file except in compliance with +- the License. You may obtain a copy of the License at +- +- http://www.apache.org/licenses/LICENSE-2.0 +- +- Unless required by applicable law or agreed to in writing, software +- distributed under the License is distributed on an "AS IS" BASIS, +- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +- See the License for the specific language governing permissions and +- limitations under the License. +--> + +# Getting Started with Hive JDBC + +## How to get the Kyuubi JDBC driver + +Kyuubi Thrift API is fully compatible with HiveServer2, so technically, it allows to use any Hive JDBC driver to connect Kyuubi Server. +But it's recommended to use [Kyuubi JDBC Driver](../client/jdbc/kyuubi_jdbc.rst), which is forked from Hive 3.1.x JDBC driver, aims to support some missing functionalities of the original Hive JDBC driver. + +The driver is available from Maven Central: + +```{parsed-literal} +<dependency> + <groupId>org.apache.kyuubi</groupId> + <artifactId>kyuubi-hive-jdbc-shaded</artifactId> + <version>{{ release }}</version> +</dependency> +``` Review Comment: I've tried different approaches, but substitution doesn't work for xml in this case. <img width="1426" height="500" alt="Image" src="https://github.com/user-attachments/assets/1ce23e8f-5f28-4875-8aa3-8dbee4bd2af6"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
