This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/main by this push:
new a7623eb KYLIN-5051, create and test docker image for kylin 4.0
a7623eb is described below
commit a7623eb811389449560fb7e0c295bbf69ee40e08
Author: Mukvin <[email protected]>
AuthorDate: Wed Sep 1 17:11:49 2021 +0800
KYLIN-5051, create and test docker image for kylin 4.0
---
docker/README-standalone.md | 36 ++++++++++------------
docker/dockerfile/standalone/Dockerfile | 15 +++++----
.../standalone/build_standalone_image.sh | 5 ++-
3 files changed, 28 insertions(+), 28 deletions(-)
diff --git a/docker/README-standalone.md b/docker/README-standalone.md
index 348a74e..adc7561 100644
--- a/docker/README-standalone.md
+++ b/docker/README-standalone.md
@@ -3,19 +3,19 @@
In order to allow users to easily try Kylin, and to facilitate developers to
verify and debug after modifying the source code. We provide the all-in-one
Kylin docker image. In this image, each service that Kylin relies on is
properly installed and deployed, including:
- Jdk 1.8
-- Hadoop 2.7.0
+- Hadoop 2.8.5
- Hive 1.2.1
-- Hbase 1.1.2 (With Zookeeper)
-- Spark 2.3.1
+- Spark 2.4.7
- Kafka 1.1.1
- MySQL 5.1.73
+- Zookeeper 3.4.6
## Quickly try Kylin with pre-built images
We have pushed the Kylin images to the [docker
hub](https://hub.docker.com/r/apachekylin/apache-kylin-standalone). You do not
need to build the image locally, just pull the image from remote (you can
browse docker hub to check the available versions):
```
-docker pull apachekylin/apache-kylin-standalone:3.1.0
+docker pull apachekylin/apache-kylin-standalone:4.0.0
```
After the pull is successful, execute "sh run_container.sh" or the following
command to start the container:
@@ -28,36 +28,34 @@ docker run -d \
-p 50070:50070 \
-p 8032:8032 \
-p 8042:8042 \
--p 16010:16010 \
---name apache-kylin-standalone \
-apachekylin/apache-kylin-standalone:3.1.0
+-p 2181:2181 \
+--name apache-kylin-4.0.0 \
+apachekylin/apache-kylin-standalone:4.0.0
```
The following services are automatically started when the container starts:
- NameNode, DataNode
- ResourceManager, NodeManager
-- HBase
-- Kafka
+- Zookeeper
+- Kafka (off by default)
- Kylin
-and run automatically `$KYLIN_HOME/bin/sample.sh `, create a
kylin_streaming_topic topic in Kafka and continue to send data to this topic.
This is to let the users start the container and then experience the batch and
streaming way to build the cube and query.
+and run automatically `$KYLIN_HOME/bin/sample.sh `, create a
kylin_streaming_topic in Kafka and continue to send data to this topic. This is
to let the users start the container and then experience the batch and
streaming way to build the cube and query.
After the container is started, we can enter the container through the `docker
exec` command. Of course, since we have mapped the specified port in the
container to the local port, we can open the pages of each service directly in
the native browser, such as:
- Kylin Web UI:
[http://127.0.0.1:7070/kylin/login](http://127.0.0.1:7070/kylin/login)
- HDFS NameNode Web UI: [http://127.0.0.1:50070](http://127.0.0.1:50070/)
- YARN ResourceManager Web UI: [http://127.0.0.1:8088](http://127.0.0.1:8088/)
-- HBase Web UI: [http://127.0.0.1:16010](http://127.0.0.1:16010/)
In the container, the relevant environment variables are as follows:
```
JAVA_HOME=/home/admin/jdk1.8.0_141
-HADOOP_HOME=/home/admin/hadoop-2.7.0
+HADOOP_HOME=/home/admin/hadoop-2.8.5
KAFKA_HOME=/home/admin/kafka_2.11-1.1.1
-SPARK_HOME=/home/admin/spark-2.3.1-bin-hadoop2.6
-HBASE_HOME=/home/admin/hbase-1.1.2
+SPARK_HOME=/home/admin/spark-2.4.7-bin-hadoop2.7
HIVE_HOME=/home/admin/apache-hive-1.2.1-bin
```
@@ -79,16 +77,16 @@ Or you can run "docker ps" to get the container id:
```
> docker ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS
NAMES
-c15d10ff6bf1 apachekylin/apache-kylin-standalone:3.1.0
"/home/admin/entrypo…" 55 minutes ago Up 55 minutes
0.0.0.0:7070->7070/tcp, 0.0.0.0:8032->8032/tcp, 0.0.0.0:8042->8042/tcp,
0.0.0.0:8088->8088/tcp, 0.0.0.0:50070->50070/tcp, 0.0.0.0:16010->16010/tcp
romantic_moser
+5a799202353b apachekylin/apache-kylin-standalone:4.0.0
"/home/admin/entrypo…" 58 minutes ago Up 43 minutes
0.0.0.0:2181->2181/tcp, 0.0.0.0:7070->7070/tcp, 0.0.0.0:8032->8032/tcp,
0.0.0.0:8042->8042/tcp, 0.0.0.0:8088->8088/tcp, 0.0.0.0:50070->50070/tcp
kylin-4.0.0
```
Then run "docker exec -it <container id> bash" to login it with bash:
```
-> docker exec -it c15d10ff6bf1 bash
-[root@c15d10ff6bf1 admin]# ls
-apache-hive-1.2.1-bin apache-maven-3.6.1 first_run
hbase-1.1.2 kafka_2.11-1.1.1
-apache-kylin-3.0.0-alpha2-bin-hbase1x entrypoint.sh hadoop-2.7.0
jdk1.8.0_141 spark-2.3.1-bin-hadoop2.6
+> docker exec -it 5a799202353b bash
+[root@5a799202353b admin]# ls
+apache-hive-1.2.1-bin apache-maven-3.6.1 first_run jdk1.8.0_141
mysql80-community-release-el7-3.noarch.rpm zookeeper-3.4.6
+apache-kylin-4.0.0-bin-spark2 entrypoint.sh hadoop-2.8.5
kafka_2.11-1.1.1 spark-2.4.7-bin-hadoop2.7
```
## Build Docker image in local
diff --git a/docker/dockerfile/standalone/Dockerfile
b/docker/dockerfile/standalone/Dockerfile
index 1d1ee3b..d12a4a8 100644
--- a/docker/dockerfile/standalone/Dockerfile
+++ b/docker/dockerfile/standalone/Dockerfile
@@ -16,16 +16,19 @@
#
# Docker image for apache kylin, based on the Hadoop image
-FROM hadoop2.7-all-in-one-for-kylin4-beta
+FROM hadoop2.7-all-in-one-for-kylin4
-ENV KYLIN_VERSION 4.0.0-beta
-ENV KYLIN_HOME /home/admin/apache-kylin-$KYLIN_VERSION-bin
+ENV RELATED_SPARK_VERSION spark2
+ENV KYLIN_VERSION 4.0.0
+ENV KYLIN_HOME
/home/admin/apache-kylin-$KYLIN_VERSION-bin-$RELATED_SPARK_VERSION
# Download Kylin
-RUN wget
https://archive.apache.org/dist/kylin/apache-kylin-$KYLIN_VERSION/apache-kylin-$KYLIN_VERSION-bin.tar.gz
\
- && tar -zxvf /home/admin/apache-kylin-$KYLIN_VERSION-bin.tar.gz \
- && rm -f /home/admin/apache-kylin-$KYLIN_VERSION-bin.tar.gz
+RUN wget
https://archive.apache.org/dist/kylin/apache-kylin-$KYLIN_VERSION/apache-kylin-$KYLIN_VERSION-bin-$RELATED_SPARK_VERSION.tar.gz
\
+ && tar -zxvf
/home/admin/apache-kylin-$KYLIN_VERSION-bin-$RELATED_SPARK_VERSION.tar.gz \
+ && rm -f
/home/admin/apache-kylin-$KYLIN_VERSION-bin-$RELATED_SPARK_VERSION.tar.gz
RUN rm -f $KYLIN_HOME/conf/kylin.properties
+# See KYLIN-5071
+RUN sed -i "s/\"kylin.engine.build-base-cuboid-enabled\":\
\"false\"/\"kylin.engine.build-base-cuboid-enabled\":\ \"true\"/g"
$KYLIN_HOME/sample_cube/template/cube_desc/kylin_sales_cube.json
COPY conf/kylin/* $KYLIN_HOME/conf/
RUN rm -f $KYLIN_HOME/bin/kylin.sh
COPY conf/bin/kylin.sh $KYLIN_HOME/bin/
diff --git a/docker/dockerfile/standalone/build_standalone_image.sh
b/docker/dockerfile/standalone/build_standalone_image.sh
index 749ebbc..8c3b96d 100755
--- a/docker/dockerfile/standalone/build_standalone_image.sh
+++ b/docker/dockerfile/standalone/build_standalone_image.sh
@@ -21,7 +21,6 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1
&& pwd )"
cd ${DIR}
echo "build image in dir "${DIR}
-
echo "start build Hadoop docker image"
-docker build -f Dockerfile_hadoop -t hadoop2.7-all-in-one-for-kylin4-beta .
-docker build -f Dockerfile -t apachekylin/apache-kylin-standalone:4.0.0-beta .
+docker build -f Dockerfile_hadoop -t hadoop2.7-all-in-one-for-kylin4 .
+docker build -f Dockerfile -t apachekylin/apache-kylin-standalone:4.0.0 .
