This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 0d85479c66f branch-4.1: [chore](docker)Support rootless Docker for
third-party environments #70457 (#70579)
0d85479c66f is described below
commit 0d85479c66f9270023d4721c73e9bb13e36d3bf8
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Nov 15 14:16:30 2026 +0800
branch-4.1: [chore](docker)Support rootless Docker for third-party
environments #70457 (#70579)
bp #70457
### Conflict resolution for branch-4.1
#70457 does not apply cleanly here: it builds on launcher wrappers that
master got from #64304, and it touches the Fluss fixture, which
branch-4.1 does not have. What differs from the original:
- **Fluss:** branch-4.1 has no Fluss docker environment, so
`docker-compose/fluss/README.md` and `fluss.env.tpl` stay deleted,
`fluss-network.sh` and `test/fluss-network-test.sh` are left out, and
the Fluss sentence in `docker/thirdparties/README.md` and the Fluss case
in `test/launcher-stop-test.sh` are dropped.
- **`run-thirdparties-docker.sh`:** branch-4.1 had no `docker_cli` /
`host_admin_cmd` wrappers, so besides the conflicted hunks, the 21
remaining `sudo docker` calls become `docker_cli` (as on master), the
Lance jar download and the Iceberg data extraction drop `sudo`, and
LakeSoul's `sudo rm -rf data` becomes `reset_data_dirs` plus `rmdir`,
the pattern #70457 uses for Iceberg. `iceberg-rest` is added to
`ALL_COMPONENTS`. Master-only code (MongoDB, Fluss, the Iceberg jar
check) is not added. The launcher has no `sudo` left.
- **`ranger_settings.env`:** takes #70457's overridable form. This also
renames branch-4.1's `RANGER_SOLAR_PORT` to `RANGER_SOLR_PORT`, the name
`ranger.yaml.tpl` already reads (master fixed it in #66770).
- **`test/launcher-stop-test.sh`:** branch-4.1's `custom_settings.env`
sets `CONTAINER_UID` unconditionally, so the test rewrites the copied
file the way CI does.
The other 25 files carry #70457's changes unchanged.
### Testing
- `bash -n` passes on every `docker/thirdparties/*.sh`.
`test/docker-command-test.sh`, `test/snapshot-hive-baseline-test.sh` and
`test/launcher-stop-test.sh` pass locally (macOS, bash 5.3, GNU getopt
as on Linux CI); the stop test fails when the `STOP` guard on Hive data
preparation is removed.
- Not run: the launcher against a real rootless Docker daemon. The
External run on this PR is the end-to-end check; External's preflight on
branch-4.1 requires this pick, and #70578 makes `TeamCity/external`
required here.
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: the conflict resolution and the adaptations above, and this
description
- Human review: not stated
---------
Co-authored-by: zhangstar333 <[email protected]>
Co-authored-by: Dongyang Li <[email protected]>Co-Authored-By: Claude
Opus 5.5 <[email protected]>
Co-authored-by: lidongyang <[email protected]>
---
docker/thirdparties/README.md | 71 +++++++
docker/thirdparties/docker-command.sh | 92 +++++++++
.../thirdparties/docker-compose/db2/db2.yaml.tpl | 5 +-
.../scripts/init-db2.sh} | 23 ++-
.../docker-compose/elasticsearch/es.yaml.tpl | 2 +
.../elasticsearch/scripts/es_init.sh | 3 +
docker/thirdparties/docker-compose/hive/README.md | 4 +-
.../thirdparties/docker-compose/hive/README_ZH.md | 4 +-
.../hive/scripts/snapshot-hive-baseline.sh | 31 ++-
.../thirdparties/docker-compose/hudi/hudi.env.tpl | 2 +-
.../thirdparties/docker-compose/hudi/hudi.yaml.tpl | 24 ++-
.../iceberg-rest/docker-compose.yaml.tpl | 10 +-
.../iceberg-rest/hdfs-rootless.yaml.tpl | 44 +++++
.../docker-compose/kafka/kafka.yaml.tpl | 5 +-
.../docker-compose/kerberos/kerberos.yaml.tpl | 6 +
.../minio/minio-RELEASE.2024-11-07.yaml.tpl | 1 -
.../docker-compose/mysql/mysql-5.7.yaml.tpl | 5 +-
.../docker-compose/oracle/oracle-11.yaml.tpl | 9 +-
.../docker-compose/polaris/docker-compose.yaml.tpl | 5 +
.../docker-compose/polaris/init-catalog.sh | 2 +
.../docker-compose/polaris/polaris_settings.env | 8 +-
.../postgresql/postgresql-14.yaml.tpl | 5 +-
.../docker-compose/ranger/ranger_settings.env | 6 +-
.../docker-compose/sqlserver/sqlserver.yaml.tpl | 20 +-
docker/thirdparties/docker-health.sh | 4 +-
docker/thirdparties/run-thirdparties-docker.sh | 217 ++++++++++-----------
docker/thirdparties/test/docker-command-test.sh | 138 +++++++++++++
docker/thirdparties/test/launcher-stop-test.sh | 79 ++++++++
.../test/snapshot-hive-baseline-test.sh | 99 ++++++++++
29 files changed, 756 insertions(+), 168 deletions(-)
diff --git a/docker/thirdparties/README.md b/docker/thirdparties/README.md
new file mode 100644
index 00000000000..507e296d937
--- /dev/null
+++ b/docker/thirdparties/README.md
@@ -0,0 +1,71 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Thirdparty Docker environments
+
+Run a component with Bash 4 or newer and Docker Compose v2:
+
+```bash
+bash run-thirdparties-docker.sh -c iceberg
+bash run-thirdparties-docker.sh -c iceberg --stop
+```
+
+## Rootless Docker on Linux
+
+Install and start the current user's rootless Docker daemon first. Its setup
+requires subordinate UID/GID ranges, `newuidmap`/`newgidmap`, and enabled
+unprivileged user namespaces. Once configured, launching these environments
+does not require host `sudo` or membership in the host `docker` group.
+
+On Linux, the launcher requires a rootless daemon and rejects rootful daemons.
+It detects the user's `$XDG_RUNTIME_DIR/docker.sock` socket unless an endpoint
+or context was explicitly selected. Docker commands always run as the current
+user; there is no automatic elevation or `DOCKER_USE_SUDO` switch.
+The endpoint can also be selected explicitly:
+
+```bash
+export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock"
+docker info --format '{{json .SecurityOptions}}' # must include name=rootless
+bash run-thirdparties-docker.sh -c iceberg
+```
+
+On macOS, the launcher uses the current user's Docker Desktop endpoint.
+
+Keep the working directory writable by the current user. The launcher cleans
+fixture files owned by subordinate container UIDs from inside the same rootless
+user namespace, using `alpine:3.20`. It does not change host ownership to root.
+
+Hive, Kerberos and Iceberg REST's built-in HDFS require Docker Engine 29.5 or
+newer for rootless host networking. Hive defaults to
`HIVE_HOST_ALIAS=localhost`
+and uses loopback for its same-host services. Iceberg REST HDFS uses
+host ports 20020, 20870, 20864, 20866 and
+20867, so its DataNode block addresses are reachable by host clients.
+
+Kerberos client configuration and keytabs remain under
+`docker-compose/kerberos/two-kerberos-hives/`; use the printed `KRB5_CONFIG`
+path for host clients. The rootless launcher does not write `/etc/hosts`,
+`/etc/krb5.conf` or `/keytabs`. The DB2 image retains its container-level
+`privileged` setting within the rootless daemon's user namespace.
+
+`--reserve-ports` changes a host sysctl and is unavailable in rootless mode.
+It is optional; an operator can reserve ports separately when required.
+Image pulls and fixture downloads still require access to their registries
+and artifact repositories. `--no-load-data` skips Hive fixture refresh;
+combine it with `--hive-mode rebuild` to validate a fresh Hive installation
+without downloading a regression-data baseline.
diff --git a/docker/thirdparties/docker-command.sh
b/docker/thirdparties/docker-command.sh
new file mode 100755
index 00000000000..ffd4533cd3e
--- /dev/null
+++ b/docker/thirdparties/docker-command.sh
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+configure_docker_access() {
+ local rootless_runtime_dir
+ local rootless_socket
+ local security_options
+
+ rootless_runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
+ rootless_socket="${rootless_runtime_dir}/docker.sock"
+
+ # An explicit endpoint or context takes precedence over socket discovery.
+ if [[ "${HOST_OS}" != "Darwin" && -z "${DOCKER_HOST+x}" \
+ && -z "${DOCKER_CONTEXT:-}" && -S "${rootless_socket}" ]]; then
+ export DOCKER_HOST="unix://${rootless_socket}"
+ fi
+
+ if ! security_options="$(docker info --format '{{json
.SecurityOptions}}')"; then
+ echo "Unable to access Docker as the current user; start the user's
rootless daemon or select its endpoint" >&2
+ return 1
+ fi
+ # Docker Desktop on macOS runs its daemon in a VM. Linux requires the
+ # user's rootless daemon; an accessible system daemon is not accepted.
+ if [[ "${HOST_OS}" != "Darwin" && "${security_options}" !=
*'"name=rootless"'* ]]; then
+ echo "This launcher requires rootless Docker on Linux; the selected
daemon is rootful" >&2
+ return 1
+ fi
+}
+
+require_rootless_host_network() {
+ local server_version
+ local major
+ local minor
+
+ if [[ "${HOST_OS}" == "Darwin" ]]; then
+ return 0
+ fi
+ server_version="$(docker_cli info --format '{{.ServerVersion}}')" || return
+ major="${server_version%%.*}"
+ minor="${server_version#*.}"
+ minor="${minor%%.*}"
+ # Older rootless host networks cannot reach ports published on the real
+ # host, which these fixtures use for PostgreSQL and HDFS RPC.
+ if ((major < 29 || (major == 29 && minor < 5))); then
+ echo "Rootless host networking requires Docker Engine 29.5 or newer
(found ${server_version})" >&2
+ return 1
+ fi
+}
+
+rootless_fixture_cmd() {
+ local fixture_dir="$1"
+ shift
+
+ # Stay in the daemon's user namespace when images have chowned bind mounts
+ # to subordinate UIDs. Mount only the fixture directory being operated on.
+ docker_cli run --rm --network none --user 0:0 \
+ --security-opt no-new-privileges \
+ --mount "type=bind,source=${fixture_dir},target=/fixture" \
+ "${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" "$@"
+}
+
+reset_rootless_data_dir() {
+ local data_dir="$1"
+
+ mkdir -p "${data_dir}"
+ rootless_fixture_cmd "${data_dir}" sh -ec '
+ find /fixture -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
+ chown 0:0 /fixture
+ chmod 755 /fixture
+ '
+}
+
+chmod_fixture_dir() {
+ local mode="$1"
+ local fixture_dir="$2"
+ rootless_fixture_cmd "${fixture_dir}" chmod -R "${mode}" /fixture
+}
diff --git a/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
b/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
index 49a09656549..c8e944d52cf 100644
--- a/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
+++ b/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
@@ -24,13 +24,14 @@ services:
- ${DOCKER_DB2_EXTERNAL_PORT}:50000
privileged: true
healthcheck:
- test: ["CMD-SHELL", "su - db2inst1 -c \"source ~/.bash_profile; db2
connect to doris && db2 'select 1 from sysibm.sysdummy1'\""]
+ test: ["CMD-SHELL", "test -f /tmp/doris_db2_initialized && su - db2inst1
-c \"source ~/.bash_profile; db2 connect to doris && db2 'select count(*) from
doris_test.sample_table'\""]
interval: 20s
timeout: 60s
retries: 30
start_period: 300s
volumes:
- - ./init:/docker-entrypoint-initdb.d
+ - ./init:/docker-entrypoint-initdb.d:ro
+ - ./scripts/init-db2.sh:/var/custom/init-db2.sh:ro
environment:
- LICENSE=accept
- DBNAME=doris
diff --git a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
b/docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
old mode 100644
new mode 100755
similarity index 52%
copy from docker/thirdparties/docker-compose/ranger/ranger_settings.env
copy to docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
index 13dd93d517d..e98334d1e97
--- a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
+++ b/docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
@@ -16,6 +16,23 @@
# specific language governing permissions and limitations
# under the License.
-export RANGER_SOLAR_PORT=8983
-export RANGER_PORT=6081
-export RANGER_MYSQL_PORT=33061
+set -euo pipefail
+rm -f /tmp/doris_db2_initialized
+
+# The IBM image executes /var/custom scripts after creating DBNAME. It does
+# not execute SQL files mounted at /docker-entrypoint-initdb.d by itself.
+su - "${DB2INSTANCE}" -c 'bash -se' <<'EOF'
+. "${HOME}/sqllib/db2profile"
+db2 connect to doris
+schema_count=$(db2 -x "SELECT COUNT(*) FROM SYSCAT.SCHEMATA WHERE
SCHEMANAME='DORIS_TEST'")
+if ((schema_count == 0)); then
+ db2 -stvf /docker-entrypoint-initdb.d/02-create-schema.sql
+ db2 -stvf /docker-entrypoint-initdb.d/03-create-table.sql
+ db2 -stvf /docker-entrypoint-initdb.d/04-insert.sql
+fi
+rows=$(db2 -x 'SELECT COUNT(*) FROM doris_test.sample_table')
+[[ "${rows}" -eq 3 ]]
+db2 connect reset
+EOF
+
+touch /tmp/doris_db2_initialized
diff --git a/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
b/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
index 83303c250a0..f838af7e829 100644
--- a/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
+++ b/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
@@ -127,6 +127,8 @@ services:
ES_8_HOST: "doris--es_8"
command: [ "sh", "-c", "/mnt/scripts/es_init.sh" ]
depends_on:
+ doris--es_5:
+ condition: service_healthy
doris--es_6:
condition: service_healthy
doris--es_7:
diff --git
a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
index 51f81ced938..d8e31323b00 100755
--- a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
+++ b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
@@ -16,6 +16,8 @@
# specific language governing permissions and limitations
# under the License.
+set -eo pipefail
+
generate_bulk_request() {
local index_name=$1
local type_value=$2
@@ -234,3 +236,4 @@ generate_bulk_request "test_keyword_array" "" "item_"
"$keyword_array_data_file"
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary
"@$keyword_array_bulk_file" -H "Content-Type: application/json"
touch /tmp/SUCCESS
+exec tail -f /dev/null
diff --git a/docker/thirdparties/docker-compose/hive/README.md
b/docker/thirdparties/docker-compose/hive/README.md
index 57c6d414b1d..65b8266217b 100644
--- a/docker/thirdparties/docker-compose/hive/README.md
+++ b/docker/thirdparties/docker-compose/hive/README.md
@@ -116,11 +116,11 @@ Relevant env vars:
After bootstrapping a clean Hive stack, stop the containers and run:
```bash
-sudo docker compose -p "${CONTAINER_UID}hive3" \
+docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down
bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
\
- "${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
+ "doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```
Then upload the resulting tarball to OSS at
`oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz`
(same convention for `hive2`).
diff --git a/docker/thirdparties/docker-compose/hive/README_ZH.md
b/docker/thirdparties/docker-compose/hive/README_ZH.md
index 8cc32162dc6..546eaf3b8e5 100644
--- a/docker/thirdparties/docker-compose/hive/README_ZH.md
+++ b/docker/thirdparties/docker-compose/hive/README_ZH.md
@@ -116,11 +116,11 @@ Hive 运行态(HDFS 数据、Postgres Metastore、模块 SHA 记录)存放
在一次完整 bootstrap 成功后,停止容器并运行:
```bash
-sudo docker compose -p "${CONTAINER_UID}hive3" \
+docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down
bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
\
- "${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
+ "doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```
然后把得到的 tarball 上传到
`oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz`(`hive2`
同理)。
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
b/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
index edbe2788dcc..e8024398a2b 100755
--- a/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
@@ -24,7 +24,7 @@ set -eo pipefail
# snapshot-hive-baseline.sh <volume_prefix> <output_path>
#
# Example:
-# bash snapshot-hive-baseline.sh doris--syt--hive3 /tmp/hive3-baseline.tar.gz
+# bash snapshot-hive-baseline.sh doris-shared-hive3
/tmp/hive3-baseline.tar.gz
#
# Prerequisites:
# - All Hive containers must be STOPPED before running this script.
@@ -33,35 +33,48 @@ set -eo pipefail
VOLUME_PREFIX="${1:?Usage: $0 <volume_prefix> <output_path>}"
OUTPUT_PATH="${2:?Usage: $0 <volume_prefix> <output_path>}"
+THIRDPARTY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." &>/dev/null &&
pwd)"
+HOST_OS="$(uname -s)"
+# shellcheck source=../../../docker-command.sh
+. "${THIRDPARTY_ROOT}/docker-command.sh"
+configure_docker_access
+
+docker_cli() {
+ docker "$@"
+}
+
echo "[snapshot] volume prefix: ${VOLUME_PREFIX}"
echo "[snapshot] output: ${OUTPUT_PATH}"
# Verify all 4 volumes exist
for suffix in namenode datanode pgdata state; do
vol="${VOLUME_PREFIX}-${suffix}"
- if ! sudo docker volume inspect "${vol}" >/dev/null 2>&1; then
+ if ! docker_cli volume inspect "${vol}" >/dev/null 2>&1; then
echo "ERROR: volume ${vol} does not exist" >&2
exit 1
fi
done
_t0=$(date +%s)
+tmp_output="$(mktemp "${OUTPUT_PATH}.XXXXXX")"
+trap 'rm -f "${tmp_output}"' EXIT
# Mount all 4 volumes read-only into a single alpine container and tar them
# in one pass. Copy only the small state volume to ephemeral container storage
# so legacy baseline.version files can be dropped from newly exported tarballs.
-sudo docker run --rm \
- -v "${VOLUME_PREFIX}-namenode:/snapshot/namenode:ro" \
- -v "${VOLUME_PREFIX}-datanode:/snapshot/datanode:ro" \
- -v "${VOLUME_PREFIX}-pgdata:/snapshot/pgdata:ro" \
+docker_cli run --rm --network none --security-opt no-new-privileges \
+ -v "${VOLUME_PREFIX}-namenode:/work/namenode:ro" \
+ -v "${VOLUME_PREFIX}-datanode:/work/datanode:ro" \
+ -v "${VOLUME_PREFIX}-pgdata:/work/pgdata:ro" \
-v "${VOLUME_PREFIX}-state:/snapshot/state:ro" \
- alpine sh -c '
+ "${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" sh -ec '
mkdir -p /work/state
cp -a /snapshot/state/. /work/state/
rm -f /work/state/baseline.version
- tar czf - -C /snapshot namenode datanode pgdata -C /work state
+ tar czf - -C /work namenode datanode pgdata state
' \
- > "${OUTPUT_PATH}"
+ > "${tmp_output}"
+mv "${tmp_output}" "${OUTPUT_PATH}"
size=$(du -h "${OUTPUT_PATH}" | cut -f1)
echo "[snapshot] done took=$(( $(date +%s) - _t0 ))s size=${size}"
diff --git a/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
b/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
index e04012010e9..69eefee256c 100644
--- a/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
+++ b/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
@@ -15,7 +15,7 @@
# limitations under the License.
#
-CONTAINER_UID=doris--
+CONTAINER_UID=${CONTAINER_UID}
HUDI_NETWORK=${CONTAINER_UID}hudi-network
# Ports exposed to host
diff --git a/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
index 94cd3adb3e0..554e77f033d 100644
--- a/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
+++ b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
@@ -27,13 +27,18 @@ services:
${CONTAINER_UID}hudi-minio:
image: doristhirdpartydocker/minio:RELEASE.2025-01-20T14-49-07Z
container_name: ${CONTAINER_UID}hudi-minio
- command: server /data --console-address ":${MINIO_CONSOLE_PORT}"
+ command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
ports:
- "${MINIO_API_PORT}:9000"
- "${MINIO_CONSOLE_PORT}:9001"
+ healthcheck:
+ test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"]
+ interval: 5s
+ timeout: 10s
+ retries: 60
networks:
- ${HUDI_NETWORK}
@@ -43,13 +48,13 @@ services:
entrypoint: |
/bin/bash -c "
set -euo pipefail
- sleep 5
mc alias set myminio http://${CONTAINER_UID}hudi-minio:9000
${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}
- mc mb --quiet myminio/${HUDI_BUCKET} || true
- mc mb --quiet myminio/${HUDI_BUCKET}-tmp || true
+ mc mb --quiet --ignore-existing myminio/${HUDI_BUCKET}
+ mc mb --quiet --ignore-existing myminio/${HUDI_BUCKET}-tmp
"
depends_on:
- - ${CONTAINER_UID}hudi-minio
+ ${CONTAINER_UID}hudi-minio:
+ condition: service_healthy
networks:
- ${HUDI_NETWORK}
@@ -123,9 +128,12 @@ services:
- ./scripts:/opt/hudi-scripts
- ./cache:/opt/hudi-cache
depends_on:
- - ${CONTAINER_UID}hudi-minio
- - ${CONTAINER_UID}hudi-minio-mc
- - ${CONTAINER_UID}hudi-metastore
+ ${CONTAINER_UID}hudi-minio:
+ condition: service_started
+ ${CONTAINER_UID}hudi-minio-mc:
+ condition: service_completed_successfully
+ ${CONTAINER_UID}hudi-metastore:
+ condition: service_started
command: ["/opt/hudi-scripts/init.sh"]
ports:
- "${SPARK_UI_PORT}:8080"
diff --git
a/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
b/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
index ee70b27e3f8..e8eef3ec2b0 100644
--- a/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
+++ b/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
@@ -170,8 +170,8 @@ services:
-
CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec
- HDFS_CONF_dfs_webhdfs_enabled=true
- HDFS_CONF_dfs_permissions_enabled=false
- - HDFS_CONF_dfs_namenode_rpc_address=hdfs-namenode:8020
- - HDFS_CONF_dfs_namenode_http_address=0.0.0.0:9870
+ - HDFS_CONF_dfs_namenode_rpc___address=hdfs-namenode:8020
+ - HDFS_CONF_dfs_namenode_http___address=0.0.0.0:9870
- HDFS_CONF_dfs_replication=1
- HDFS_CONF_dfs_namenode_name_dir=/hadoop/dfs/name
-
HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
@@ -226,6 +226,12 @@ services:
depends_on:
- hdfs-namenode
- hdfs-datanode
+ healthcheck:
+ # This image has bash but does not include curl or wget.
+ test: ["CMD", "bash", "-ec", "exec 3<>/dev/tcp/127.0.0.1/8181; printf
'GET /v1/config HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n' >&3; head -n 1 <&3 |
grep -q '^HTTP/1.[01] 200 '"]
+ interval: 10s
+ timeout: 5s
+ retries: 5
networks:
${CONTAINER_UID}iceberg-rest:
diff --git
a/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
b/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
new file mode 100644
index 00000000000..2c85f826c35
--- /dev/null
+++ b/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Use distinct real host ports so HDFS block locations work for host clients
+# as well as the REST catalog containers. Requires rootless Docker >= 29.5.
+services:
+ hdfs-namenode:
+ network_mode: host
+ networks: !reset []
+ ports: !reset []
+ environment:
+ - CORE_CONF_fs_defaultFS=hdfs://${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ -
HDFS_CONF_dfs_namenode_rpc___address=${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ -
HDFS_CONF_dfs_namenode_http___address=0.0.0.0:${HDFS_NAMENODE_WEBUI_PORT}
+ healthcheck:
+ test: ["CMD", "curl", "-f",
"http://127.0.0.1:${HDFS_NAMENODE_WEBUI_PORT}/"]
+
+ hdfs-datanode:
+ network_mode: host
+ networks: !reset []
+ ports: !reset []
+ environment:
+ - CORE_CONF_fs_defaultFS=hdfs://${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ - HDFS_CONF_dfs_datanode_hostname=${LOCAL_IP}
+ - HDFS_CONF_dfs_datanode_address=0.0.0.0:20866
+ - HDFS_CONF_dfs_datanode_ipc_address=0.0.0.0:20867
+ - HDFS_CONF_dfs_datanode_http_address=0.0.0.0:${HDFS_DATANODE_WEBUI_PORT}
+ - SERVICE_PRECONDITION=${LOCAL_IP}:${HDFS_NAMENODE_WEBUI_PORT}
+ healthcheck:
+ test: ["CMD", "curl", "-f",
"http://127.0.0.1:${HDFS_DATANODE_WEBUI_PORT}/"]
diff --git a/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
b/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
index e3322ce2ff6..777f744b8b6 100644
--- a/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
@@ -31,8 +31,7 @@ services:
ports:
- ${DOCKER_ZOOKEEPER_EXTERNAL_PORT}:2181
healthcheck:
- #
https://github.com/bitnami/charts/blob/62399ed3863e70775d66f9581e28129026a86e5d/bitnami/zookeeper/templates/statefulset.yaml#L404
- test: /bin/bash -ec ZOO_HC_TIMEOUT=10
/opt/bitnami/scripts/zookeeper/healthcheck.sh
+ test: ["CMD", "/opt/bitnami/zookeeper/bin/zkServer.sh", "status"]
start_period: 10s
interval: 5s
timeout: 60s
@@ -63,7 +62,5 @@ services:
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:19193
- KAFKA_ZOOKEEPER_CONNECT=doris--zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- volumes:
- - /var/run/docker.sock:/var/run/docker.sock
networks:
- doris--kafka--network
diff --git a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
index cf1dc94867c..2a749642d32 100644
--- a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
@@ -24,6 +24,12 @@ x-kerberos-image: &kerberos-image
x-kerberos-service: &kerberos-service
<<: *kerberos-image
network_mode: "host"
+ extra_hosts:
+ - "hadoop-master:${IP_HOST}"
+ - "hadoop-master-2:${IP_HOST}"
+ # Hadoop's Kerberos resolver performs absolute (trailing-dot) lookups.
+ - "hadoop-master.:${IP_HOST}"
+ - "hadoop-master-2.:${IP_HOST}"
mem_limit: 1536m
stop_grace_period: 30s
diff --git
a/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
b/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
index 81ddd8ba027..6dfad86b97a 100644
--- a/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
+++ b/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
@@ -23,7 +23,6 @@ services:
restart: always
ports:
- ${DOCKER_MINIO_EXTERNAL_PORT}:9000
- privileged: true
healthcheck:
test: [ "CMD", "bash", "-c", "mc ping --count 1 local" ]
interval: 20s
diff --git a/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
b/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
index 9fe7b1a38ef..d77c52394ed 100644
--- a/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
+++ b/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
@@ -32,6 +32,9 @@ services:
- bash
- -c
- |
+ set -e
+ mkdir -p /etc/mysql/certs
+ cp /etc/mysql/certs-source/* /etc/mysql/certs/
chown mysql:mysql /etc/mysql/certs/*
chmod 600 /etc/mysql/certs/server.key
chmod 644 /etc/mysql/certs/server.crt /etc/mysql/certs/root.crt
@@ -52,7 +55,7 @@ services:
- ./data/:/var/lib/mysql
- ./init:/docker-entrypoint-initdb.d
- ./my.cnf:/etc/mysql/conf.d/my.cnf
- - ./certs:/etc/mysql/certs
+ - ./certs:/etc/mysql/certs-source:ro
networks:
- doris--mysql_57
diff --git a/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
b/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
index 09314c9dc23..087833bb649 100644
--- a/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
+++ b/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
@@ -23,9 +23,14 @@ services:
restart: always
ports:
- ${DOCKER_ORACLE_EXTERNAL_PORT}:1521
- privileged: true
healthcheck:
- test: [ "CMD", "bash", "-c", "echo 'SELECT 1 FROM doris_test.deadline;'
| ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
/u01/app/oracle/product/11.2.0/xe/bin/sqlplus -s DORIS_TEST/123456@localhost" ]
+ test:
+ - CMD-SHELL
+ - >-
+ printf '%s\n' 'WHENEVER SQLERROR EXIT FAILURE' 'WHENEVER OSERROR
EXIT FAILURE'
+ 'SELECT COUNT(*) FROM doris_test.deadline;' 'EXIT;'
+ | ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
+ /u01/app/oracle/product/11.2.0/xe/bin/sqlplus -L -s
DORIS_TEST/123456@localhost
interval: 20s
timeout: 60s
retries: 120
diff --git a/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
b/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
index 98124890872..78c941cf03b 100644
--- a/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
+++ b/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
@@ -113,6 +113,11 @@ services:
volumes:
- ./init-catalog.sh:/init-catalog.sh:ro
entrypoint: ["/bin/sh","-c","/init-catalog.sh"]
+ healthcheck:
+ test: ["CMD-SHELL", "test -f /tmp/polaris_initialized"]
+ interval: 5s
+ timeout: 3s
+ retries: 60
networks:
- ${CONTAINER_UID}polaris
diff --git a/docker/thirdparties/docker-compose/polaris/init-catalog.sh
b/docker/thirdparties/docker-compose/polaris/init-catalog.sh
index 124e41845be..6e0b5a9fb0b 100755
--- a/docker/thirdparties/docker-compose/polaris/init-catalog.sh
+++ b/docker/thirdparties/docker-compose/polaris/init-catalog.sh
@@ -142,4 +142,6 @@ fi
echo "[polaris-init] Permissions setup completed."
echo "[polaris-init] Done."
+touch /tmp/polaris_initialized
+exec tail -f /dev/null
diff --git a/docker/thirdparties/docker-compose/polaris/polaris_settings.env
b/docker/thirdparties/docker-compose/polaris/polaris_settings.env
index 70ba1a96b93..2242db926d4 100644
--- a/docker/thirdparties/docker-compose/polaris/polaris_settings.env
+++ b/docker/thirdparties/docker-compose/polaris/polaris_settings.env
@@ -17,15 +17,15 @@
# under the License.
# Polaris uses 20181-20182 for S3
-export POLARIS_S3_PORT=20181
-export POLARIS_S3_ADMIN_PORT=20182
+export POLARIS_S3_PORT="${POLARIS_S3_PORT:-20181}"
+export POLARIS_S3_ADMIN_PORT="${POLARIS_S3_ADMIN_PORT:-20182}"
# Polaris authentication
export POLARIS_BOOTSTRAP_PASSWORD="secret123"
# MinIO (S3-compatible) configuration for local development
-export MINIO_API_PORT=20001
-export MINIO_CONSOLE_PORT=20002
+export MINIO_API_PORT="${POLARIS_MINIO_API_PORT:-20001}"
+export MINIO_CONSOLE_PORT="${POLARIS_MINIO_CONSOLE_PORT:-20002}"
export MINIO_ACCESS_KEY="admin"
export MINIO_SECRET_KEY="password"
export MINIO_BUCKET="warehouse"
diff --git
a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
index 54758285290..a8d1233c1f1 100644
--- a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
+++ b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
@@ -29,6 +29,9 @@ services:
- bash
- -c
- |
+ set -e
+ mkdir -p /var/lib/postgresql/certs
+ cp /var/lib/postgresql/certs-source/* /var/lib/postgresql/certs/
chmod 600 /var/lib/postgresql/certs/server.key
chmod 644 /var/lib/postgresql/certs/server.crt
chmod 644 /var/lib/postgresql/certs/root.crt
@@ -59,7 +62,7 @@ services:
volumes:
- ./data/data/:/var/lib/postgresql/data/
- ./init:/docker-entrypoint-initdb.d
- - ./certs:/var/lib/postgresql/certs
+ - ./certs:/var/lib/postgresql/certs-source:ro
networks:
- doris--postgres
diff --git a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
b/docker/thirdparties/docker-compose/ranger/ranger_settings.env
index 13dd93d517d..08c11de6495 100644
--- a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
+++ b/docker/thirdparties/docker-compose/ranger/ranger_settings.env
@@ -16,6 +16,6 @@
# specific language governing permissions and limitations
# under the License.
-export RANGER_SOLAR_PORT=8983
-export RANGER_PORT=6081
-export RANGER_MYSQL_PORT=33061
+export RANGER_SOLR_PORT="${RANGER_SOLR_PORT:-8983}"
+export RANGER_PORT="${RANGER_PORT:-6081}"
+export RANGER_MYSQL_PORT="${RANGER_MYSQL_PORT:-33061}"
diff --git a/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
b/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
index c2195d2f022..b14707155de 100644
--- a/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
+++ b/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
@@ -23,7 +23,7 @@ services:
ports:
- ${DOCKER_SQLSERVER_EXTERNAL_PORT}:1433
healthcheck:
- test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-Usa", "-PDoris123456",
"-Q", "select 1"]
+ test: ["CMD-SHELL", "test -f /tmp/doris_sqlserver_initialized &&
/opt/mssql-tools/bin/sqlcmd -Usa -PDoris123456 -b -Q 'SELECT 1 FROM
doris_test.dbo.student'"]
interval: 5s
timeout: 30s
retries: 120
@@ -33,26 +33,26 @@ services:
- /bin/bash
- -c
- |
+ set -e
+ rm -f /tmp/doris_sqlserver_initialized
# Launch MSSQL and send to background
/opt/mssql/bin/sqlservr &
+ sqlserver_pid=$$!
+ trap 'kill -TERM "$$sqlserver_pid"; wait "$$sqlserver_pid"' TERM INT
# Wait for it to be available
echo "Waiting for MS SQL to be available ⏳"
- /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -V1 -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"
- is_up=$$?
- while [ $$is_up -ne 0 ] ; do
- echo -e $$(date)
- /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -V1 -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"
- is_up=$$?
+ until /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -b -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"; do
+ kill -0 "$$sqlserver_pid"
sleep 5
done
# Run every script in /scripts
# TODO set a flag so that this is only done once on creation,
# and not every time the container runs
for foo in /docker-entrypoint-initdb.d/*.sql
- do /opt/mssql-tools/bin/sqlcmd -U sa -P Doris123456 -l 30 -e -i $$foo
+ do /opt/mssql-tools/bin/sqlcmd -U sa -P Doris123456 -b -l 30 -e -i
"$$foo"
done
- # So that the container doesn't shut down, sleep this thread
- sleep infinity
+ touch /tmp/doris_sqlserver_initialized
+ wait "$$sqlserver_pid"
restart: always
environment:
diff --git a/docker/thirdparties/docker-health.sh
b/docker/thirdparties/docker-health.sh
index 9148465a54d..7eca1e6f30a 100644
--- a/docker/thirdparties/docker-health.sh
+++ b/docker/thirdparties/docker-health.sh
@@ -18,12 +18,12 @@
docker_container_state_cmd() {
local container_name="$1"
- sudo docker inspect --format '{{.State.Status}}|{{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_name}"
+ docker_cli inspect --format '{{.State.Status}}|{{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_name}"
}
docker_container_env_cmd() {
local container_name="$1"
- sudo docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}'
"${container_name}"
+ docker_cli inspect --format '{{range .Config.Env}}{{println .}}{{end}}'
"${container_name}"
}
docker_container_ready() {
diff --git a/docker/thirdparties/run-thirdparties-docker.sh
b/docker/thirdparties/run-thirdparties-docker.sh
index 14babf73026..6402e4a379d 100755
--- a/docker/thirdparties/run-thirdparties-docker.sh
+++ b/docker/thirdparties/run-thirdparties-docker.sh
@@ -23,6 +23,13 @@
set -eo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+HOST_OS="$(uname -s)"
+
+. "${ROOT}/docker-command.sh"
+
+docker_cli() {
+ docker "$@"
+}
. "${ROOT}/custom_settings.env"
. "${ROOT}/juicefs-helpers.sh"
@@ -38,7 +45,7 @@ Usage: $0 <options>
-c mysql start MySQL
-c mysql,hive3 start MySQL and Hive3
--stop stop the specified components
- --reserve-ports reserve host ports by setting
'net.ipv4.ip_local_reserved_ports' to avoid port already bind error
+ --reserve-ports unsupported; configure host port reservations
separately
--no-load-data do not load data into the components
--load-parallel <num> set the parallel number to load data, default is
the 50% of CPU cores
--hive-mode <mode> hive startup mode: fast, refresh, rebuild
@@ -50,7 +57,7 @@ Usage: $0 <options>
exit 1
}
DEFAULT_COMPONENTS="mysql,es,hive2,hive3,pg,oracle,sqlserver,clickhouse,mariadb,iceberg,hudi,db2,oceanbase,kerberos,minio"
-ALL_COMPONENTS="${DEFAULT_COMPONENTS},kafka,lakesoul,ranger,polaris"
+ALL_COMPONENTS="${DEFAULT_COMPONENTS},iceberg-rest,kafka,lakesoul,ranger,polaris"
COMPONENTS=$2
HELP=0
STOP=0
@@ -168,8 +175,10 @@ if [[ "${CONTAINER_UID}"x == "doris--"x ]]; then
exit 1
fi
+configure_docker_access
+
if [[ -z "${HIVE_HOST_ALIAS:-}" ]]; then
- export HIVE_HOST_ALIAS="hadoop-master-${HIVE_SHARED_ID}"
+ export HIVE_HOST_ALIAS=localhost
fi
echo "Components are: ${COMPONENTS}"
@@ -326,10 +335,8 @@ reserve_ports() {
return
fi
- if [[ "${RESERVED_PORTS}"x != ""x ]]; then
- echo "Reserve ports: ${RESERVED_PORTS}"
- sudo sysctl -w net.ipv4.ip_local_reserved_ports="${RESERVED_PORTS}"
- fi
+ echo "--reserve-ports changes host sysctl settings; configure it
separately as an operator" >&2
+ return 1
}
JFS_META_FORMATTED=0
@@ -435,9 +442,9 @@ ensure_juicefs_meta_database() {
return 0
fi
- mysql_container=$(sudo docker ps --format '{{.Names}}' | grep -E
"(^|-)${CONTAINER_UID}mysql_57(-[0-9]+)?$" | head -n 1 || true)
+ mysql_container=$(docker_cli ps --format '{{.Names}}' | grep -E
"(^|-)${CONTAINER_UID}mysql_57(-[0-9]+)?$" | head -n 1 || true)
if [[ -n "${mysql_container}" ]]; then
- if sudo docker exec "${mysql_container}" \
+ if docker_cli exec "${mysql_container}" \
mysql -uroot -p123456 -e "CREATE DATABASE IF NOT EXISTS
\`${meta_db}\`;" >/dev/null 2>&1; then
return 0
fi
@@ -462,16 +469,16 @@ ensure_juicefs_meta_database() {
)
for pg_container in "${pg_candidates[@]}"; do
- if ! sudo docker ps --format '{{.Names}}' | grep -Fxq
"${pg_container}"; then
+ if ! docker_cli ps --format '{{.Names}}' | grep -Fxq
"${pg_container}"; then
continue
fi
- if sudo docker exec "${pg_container}" \
+ if docker_cli exec "${pg_container}" \
psql -U postgres -d postgres -tAc "SELECT 1 FROM pg_database
WHERE datname='${meta_db}'" | grep -q '^1$'; then
return 0
fi
- if sudo docker exec "${pg_container}" \
+ if docker_cli exec "${pg_container}" \
psql -U postgres -d postgres -c "CREATE DATABASE
\"${meta_db}\";" >/dev/null 2>&1; then
return 0
fi
@@ -551,8 +558,8 @@ prepare_juicefs_meta_for_hive() {
fi
local bucket_dir="${JFS_BUCKET_DIR:-/tmp/jfs-bucket}"
- sudo mkdir -p "${bucket_dir}"
- sudo chmod 777 "${bucket_dir}"
+ mkdir -p "${bucket_dir}"
+ chmod_fixture_dir 777 "${bucket_dir}"
# For local docker metadata DSNs (mysql/postgresql), ensure metadata
database exists.
ensure_juicefs_meta_database "${jfs_meta}"
@@ -568,7 +575,8 @@ prepare_juicefs_meta_for_hive() {
# and will cause "juicefs format" to fail with "Storage ... is not empty".
if [[ -d "${bucket_dir}" ]]; then
echo "Cleaning stale JuiceFS bucket directory: ${bucket_dir}"
- sudo rm -rf "${bucket_dir:?}"/*
+ reset_data_dirs "${bucket_dir:?}"
+ chmod_fixture_dir 777 "${bucket_dir}"
fi
if ! run_juicefs_cli \
@@ -597,9 +605,9 @@ compose_cmd() {
shift 2
if [[ -n "${env_file}" ]]; then
- sudo docker compose -f "${compose_file}" --env-file "${env_file}" "$@"
+ docker_cli compose -f "${compose_file}" --env-file "${env_file}" "$@"
else
- sudo docker compose -f "${compose_file}" "$@"
+ docker_cli compose -f "${compose_file}" "$@"
fi
}
@@ -623,8 +631,7 @@ reset_data_dirs() {
local data_dir
for data_dir in "$@"; do
- sudo mkdir -p "${data_dir}"
- sudo rm -rf "${data_dir:?}"/*
+ reset_rootless_data_dir "${data_dir}"
done
}
@@ -833,7 +840,7 @@ dump_start_failure() {
fi
echo "===== unhealthy containers =====" >&2
- sudo docker ps -a --filter 'health=unhealthy' --format '{{.Names}} |
{{.Image}} | {{.Status}}' >&2 || true
+ docker_cli ps -a --filter 'health=unhealthy' --format '{{.Names}} |
{{.Image}} | {{.Status}}' >&2 || true
}
print_started_summary() {
@@ -862,7 +869,7 @@ print_started_summary() {
echo " containers:"
while read -r container_id; do
[[ -n "${container_id}" ]] || continue
- sudo docker inspect --format '{{.Name}} | {{.Config.Image}} |
{{.State.Status}} | health={{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_id}" \
+ docker_cli inspect --format '{{.Name}} | {{.Config.Image}} |
{{.State.Status}} | health={{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_id}" \
2>/dev/null || true
done <<<"${compose_ids}"
fi
@@ -912,22 +919,16 @@ start_es() {
register_stack_metadata "es"
"${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env"
compose_down_stack "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" --remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es6/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es6/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es7/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es7/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es8/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es8/*
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/data
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es6/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es6/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es7/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es7/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es8/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es8/*
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/logs
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/config
- compose_cmd "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" up -d --remove-orphans
+ local version
+ for version in 5 6 7 8; do
+ reset_data_dirs \
+ "${ROOT}/docker-compose/elasticsearch/data/es${version}" \
+ "${ROOT}/docker-compose/elasticsearch/logs/es${version}"
+ done
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/data"
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/logs"
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/config"
+ compose_cmd "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" up -d --wait --remove-orphans
fi
}
@@ -1017,7 +1018,7 @@ start_kafka() {
for topic in "${topics[@]}"; do
echo "Creating kafka topic '${topic}' for ${container_id}"
- sudo docker exec "${container_id}" bash -c
"/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server
'${ip_host}:19193' --topic '${topic}'"
+ docker_cli exec "${container_id}" bash -c
"/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server
'${ip_host}:19193' --topic '${topic}'"
done
}
@@ -1028,7 +1029,7 @@ start_kafka() {
local attempt
for attempt in {1..30}; do
- if sudo docker exec "${container_id}" bash -c
"/opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server
'${ip_host}:19193'" >/dev/null 2>&1; then
+ if docker_cli exec "${container_id}" bash -c
"/opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server
'${ip_host}:19193'" >/dev/null 2>&1; then
return 0
fi
sleep 2
@@ -1039,7 +1040,7 @@ start_kafka() {
}
if [[ "${STOP}" -ne 1 ]]; then
- compose_up_stack "${ROOT}/docker-compose/kafka/kafka.yaml"
"${ROOT}/docker-compose/kafka/kafka.env" --build --remove-orphans -d
+ compose_up_stack "${ROOT}/docker-compose/kafka/kafka.yaml"
"${ROOT}/docker-compose/kafka/kafka.env" --build --remove-orphans -d --wait
wait_for_kafka_ready "${KAFKA_CONTAINER_ID}" "${IP_HOST}"
create_kafka_topics "${KAFKA_CONTAINER_ID}" "${IP_HOST}"
fi
@@ -1070,8 +1071,8 @@ ensure_hive_volumes() {
local prefix="$1"
local suffix
for suffix in "${HIVE_VOLUME_SUFFIXES[@]}"; do
- if ! sudo docker volume inspect "${prefix}-${suffix}" >/dev/null 2>&1;
then
- sudo docker volume create "${prefix}-${suffix}" >/dev/null
+ if ! docker_cli volume inspect "${prefix}-${suffix}" >/dev/null 2>&1;
then
+ docker_cli volume create "${prefix}-${suffix}" >/dev/null
fi
done
}
@@ -1080,13 +1081,13 @@ reset_hive_volumes() {
local prefix="$1"
local suffix
for suffix in "${HIVE_VOLUME_SUFFIXES[@]}"; do
- sudo docker volume rm -f "${prefix}-${suffix}" >/dev/null 2>&1 || true
+ docker_cli volume rm -f "${prefix}-${suffix}" >/dev/null 2>&1 || true
done
}
hive_volume_is_populated() {
local prefix="$1"
- sudo docker run --rm \
+ docker_cli run --rm \
-v "${prefix}-namenode:/vol:ro" \
alpine test -f /vol/current/VERSION 2>/dev/null
}
@@ -1186,7 +1187,7 @@ maybe_restore_baseline_to_volumes() {
echo "[baseline] restoring volumes from extracted baseline cache..."
local _t0
_t0=$(date +%s)
- sudo docker run --rm \
+ docker_cli run --rm \
-v "${extracted_dir}:/baseline:ro" \
-v "${prefix}-namenode:/restore/namenode" \
-v "${prefix}-datanode:/restore/datanode" \
@@ -1230,32 +1231,14 @@ hive_metastore_container_for() {
ensure_hosts_alias() {
local alias_name="$1"
local alias_ip="$2"
- local tmp_hosts
- local sudo_cmd=()
-
- if [[ "$(id -u)" -ne 0 ]]; then
- sudo_cmd=(sudo)
- fi
-
- tmp_hosts="$(mktemp)"
- "${sudo_cmd[@]}" chmod a+w /etc/hosts
- awk -v alias_name="${alias_name}" '
- {
- keep = 1
- for (i = 2; i <= NF; ++i) {
- if ($i == alias_name) {
- keep = 0
- break
- }
- }
- if (keep) {
- print
- }
- }
- ' /etc/hosts >"${tmp_hosts}"
- printf "%s %s\n" "${alias_ip}" "${alias_name}" >>"${tmp_hosts}"
- "${sudo_cmd[@]}" cp "${tmp_hosts}" /etc/hosts
- rm -f "${tmp_hosts}"
+ if [[ "${alias_name}" == "localhost" && "${alias_ip}" == "127.0.0.1" ]];
then
+ return 0
+ fi
+ if getent ahostsv4 "${alias_name}" | awk -v ip="${alias_ip}" '$1 == ip {
found = 1 } END { exit !found }'; then
+ return 0
+ fi
+ echo "${alias_name} must already resolve to ${alias_ip}; set
HIVE_HOST_ALIAS=localhost and IP_HOST=127.0.0.1 for local fixtures" >&2
+ return 1
}
render_hive_compose() {
@@ -1277,7 +1260,7 @@ render_hive_compose() {
hive_compose_cmd() {
local hive_version="$1"
- sudo docker compose -p "${CONTAINER_UID}${hive_version}" -f
"$(hive_compose_file_for "${hive_version}")" --env-file "$(hive_env_file_for
"${hive_version}")" "${@:2}"
+ docker_cli compose -p "${CONTAINER_UID}${hive_version}" -f
"$(hive_compose_file_for "${hive_version}")" --env-file "$(hive_env_file_for
"${hive_version}")" "${@:2}"
}
exec_hive_script() {
@@ -1289,7 +1272,7 @@ exec_hive_script() {
# -i: forward SIGINT/SIGTERM into container so Ctrl+C kills the
in-container script
# instead of leaving an orphan that keeps mutating state.
# stdbuf -oL -eL: line-buffer output so progress reaches the host log in
real time.
- sudo docker exec -i \
+ docker_cli exec -i \
-e HIVE_BOOTSTRAP_GROUPS="${HIVE_BOOTSTRAP_GROUPS}" \
-e LOAD_PARALLEL="${LOAD_PARALLEL}" \
-e HIVE_HQL_PARALLEL="${HIVE_HQL_PARALLEL}" \
@@ -1334,6 +1317,12 @@ start_hive_stack() {
local volume_prefix
local baseline_restore_result="missing"
+ # localhost metadata and host-network services must use loopback on both
+ # sides, even when other components advertise the host's LAN address.
+ if [[ "${HIVE_HOST_ALIAS}" == "localhost" ]]; then
+ export IP_HOST=127.0.0.1
+ fi
+
export HIVE_BOOTSTRAP_GROUPS="$(hive_bootstrap_groups_for
"${hive_version}")"
echo "${hive_version} selected bootstrap files: ${HIVE_BOOTSTRAP_GROUPS}"
@@ -1342,16 +1331,17 @@ start_hive_stack() {
export HIVE_VOLUME_PREFIX="${volume_prefix}"
log_hive_volumes "${hive_version}" "${volume_prefix}"
- # Keep a stable hostname in metastore/HDFS metadata while allowing the
- # backing host IP to change across restarts.
- ensure_hosts_alias "${HIVE_HOST_ALIAS}" "${IP_HOST}"
-
if [[ "${STOP}" -eq 1 ]]; then
render_hive_compose "${hive_version}"
hive_compose_cmd "${hive_version}" down
return 0
fi
+ # Keep a stable hostname in metastore/HDFS metadata while allowing the
+ # backing host IP to change across restarts. Stopping needs no DNS setup.
+ require_rootless_host_network
+ ensure_hosts_alias "${HIVE_HOST_ALIAS}" "${IP_HOST}"
+
# refresh/rebuild: tear down the stack and clear volumes first.
# fast: keep existing volumes and only restore the baseline when they are
empty.
if [[ "${HIVE_MODE}" == "rebuild" || "${HIVE_MODE}" == "refresh" ]]; then
@@ -1406,9 +1396,9 @@ start_iceberg() {
cd "${ICEBERG_DIR}" || exit 1
rm -f iceberg_data*.zip
wget -P "${ROOT}/docker-compose/iceberg"
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip"
- sudo unzip iceberg_data_spark40_paimon142.zip
- sudo mv iceberg_data data
- sudo rm -rf iceberg_data_spark40_paimon142.zip
+ unzip iceberg_data_spark40_paimon142.zip
+ mv iceberg_data data
+ rm -rf iceberg_data_spark40_paimon142.zip
)
else
echo "${ICEBERG_DIR}/data exists, continue !"
@@ -1416,9 +1406,9 @@ start_iceberg() {
if [[ ! -s "${LANCE_SPARK_JAR_PATH}" ]]; then
echo "Downloading ${LANCE_SPARK_JAR}"
- sudo mkdir -p "$(dirname "${LANCE_SPARK_JAR_PATH}")"
- sudo wget -O "${LANCE_SPARK_JAR_PATH}.tmp" "${LANCE_SPARK_JAR_URL}"
- sudo mv "${LANCE_SPARK_JAR_PATH}.tmp" "${LANCE_SPARK_JAR_PATH}"
+ mkdir -p "$(dirname "${LANCE_SPARK_JAR_PATH}")"
+ wget -O "${LANCE_SPARK_JAR_PATH}.tmp" "${LANCE_SPARK_JAR_URL}"
+ mv "${LANCE_SPARK_JAR_PATH}.tmp" "${LANCE_SPARK_JAR_PATH}"
fi
compose_up_stack "${ROOT}/docker-compose/iceberg/iceberg.yaml"
"${ROOT}/docker-compose/iceberg/iceberg.env" -d --wait
@@ -1437,7 +1427,7 @@ start_hudi() {
. "${HUDI_DIR}"/hudi.env
set +a
envsubst <"${HUDI_DIR}"/hudi.yaml.tpl >"${HUDI_DIR}"/hudi.yaml
- sudo chmod +x "${HUDI_DIR}"/scripts/init.sh
+ chmod +x "${HUDI_DIR}"/scripts/init.sh
register_stack_metadata "hudi" "${HUDI_DIR}/hudi.yaml"
"${HUDI_DIR}/hudi.env"
compose_down_stack "${HUDI_DIR}/hudi.yaml" "${HUDI_DIR}/hudi.env"
--remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
@@ -1461,7 +1451,8 @@ start_lakesoul() {
sed -i "s/doris--/${CONTAINER_UID}/g"
"${ROOT}"/docker-compose/lakesoul/lakesoul.yaml
register_stack_metadata "lakesoul"
"${ROOT}/docker-compose/lakesoul/lakesoul.yaml" ""
compose_cmd "${ROOT}/docker-compose/lakesoul/lakesoul.yaml" "" down
--remove-orphans
- sudo rm -rf "${ROOT}"/docker-compose/lakesoul/data
+ reset_data_dirs "${ROOT}/docker-compose/lakesoul/data"
+ rmdir "${ROOT}/docker-compose/lakesoul/data"
if [[ "${STOP}" -ne 1 ]]; then
echo "PREPARE_LAKESOUL_DATA"
compose_cmd "${ROOT}/docker-compose/lakesoul/lakesoul.yaml" "" up -d
@@ -1496,18 +1487,18 @@ dump_kerberos_container_state() {
local container="$1"
echo "===== ${container} state =====" >&2
- sudo docker inspect --format \
+ docker_cli inspect --format \
'status={{.State.Status}} running={{.State.Running}}
exitCode={{.State.ExitCode}} oomKilled={{.State.OOMKilled}}
error={{.State.Error}}' \
"${container}" >&2 || true
echo "===== ${container} logs (tail -200) =====" >&2
- sudo docker logs --tail 200 "${container}" >&2 || true
+ docker_cli logs --tail 200 "${container}" >&2 || true
}
cleanup_kerberos_ready_wait() {
local wait_pid="$1"
[[ -n "${wait_pid}" ]] || return 0
- sudo kill -TERM -- "-${wait_pid}" >/dev/null 2>&1 || true
+ kill -TERM -- "-${wait_pid}" >/dev/null 2>&1 || true
wait "${wait_pid}" >/dev/null 2>&1 || true
}
@@ -1517,7 +1508,7 @@ wait_for_kerberos_ready() {
local wait_status=0
echo "Waiting for ${container} readiness event"
- setsid sudo timeout --signal=TERM --kill-after=5s 20m bash -c '
+ setsid timeout --signal=TERM --kill-after=5s 20m bash -c '
log_dir=$(mktemp -d)
mkfifo "${log_dir}/container.log"
docker logs --follow "$1" >"${log_dir}/container.log" 2>&1 &
@@ -1567,7 +1558,7 @@ validate_kerberos_container() {
local container="$1"
echo "Running one-shot readiness validation for ${container}"
- if ! sudo docker exec "${container}" /opt/doris/health.sh; then
+ if ! docker_cli exec "${container}" /opt/doris/health.sh; then
echo "ERROR: one-shot readiness validation failed for ${container}" >&2
dump_kerberos_container_state "${container}"
return 1
@@ -1642,21 +1633,12 @@ start_kerberos() {
envsubst <"${KERBEROS_DIR}/conf/${config}.tpl"
>"${KERBEROS_DIR}/conf/kerberos${i}/${config}"
done
done
- sudo chmod a+w /etc/hosts
- if ! awk -v ip="${IP_HOST}" '$1 == ip && $2 == "hadoop-master" { found = 1
} END { exit !found }' /etc/hosts; then
- sudo sed -i "1i${IP_HOST} hadoop-master" /etc/hosts
- fi
- if ! awk -v ip="${IP_HOST}" '$1 == ip && $2 == "hadoop-master-2" { found =
1 } END { exit !found }' /etc/hosts; then
- sudo sed -i "1i${IP_HOST} hadoop-master-2" /etc/hosts
- fi
register_stack_metadata "kerberos" "${KERBEROS_DIR}/kerberos.yaml" ""
compose_cmd "${KERBEROS_DIR}/kerberos.yaml" "" down --remove-orphans
- sudo rm -rf "${KERBEROS_DIR}/data"
if [[ "${STOP}" -ne 1 ]]; then
+ require_rootless_host_network
echo "PREPARE KERBEROS DATA"
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.keytab
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.jks
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.conf
+ reset_data_dirs "${KERBEROS_DIR}/data"
"${KERBEROS_DIR}/two-kerberos-hives"
compose_cmd "${KERBEROS_DIR}/kerberos.yaml" "" up --build
--remove-orphans -d
trap 'cleanup_kerberos_readiness_jobs "${readiness_pids[@]}"' EXIT
trap 'exit 143' TERM
@@ -1678,9 +1660,13 @@ start_kerberos() {
for container in "${containers[@]}"; do
validate_kerberos_container "${container}"
done
- sudo ln -sfn "${KERBEROS_DIR}/two-kerberos-hives" /keytabs
- sudo cp "${KERBEROS_DIR}/common/conf/doris-krb5.conf"
/keytabs/krb5.conf
- sudo cp "${KERBEROS_DIR}/common/conf/doris-krb5.conf" /etc/krb5.conf
+ sed -e "s/kdc = hadoop-master:/kdc = ${IP_HOST}:/" \
+ -e "s/kdc = hadoop-master-2:/kdc = ${IP_HOST}:/" \
+ -e "s#FILE:/var/log/#FILE:${KERBEROS_DIR}/two-kerberos-hives/#g" \
+ "${KERBEROS_DIR}/common/conf/doris-krb5.conf" \
+ >"${KERBEROS_DIR}/two-kerberos-hives/krb5.conf"
+ echo "Kerberos client files: ${KERBEROS_DIR}/two-kerberos-hives"
+ echo "For host clients: export
KRB5_CONFIG=${KERBEROS_DIR}/two-kerberos-hives/krb5.conf"
sleep 2
fi
}
@@ -1709,11 +1695,8 @@ start_polaris() {
register_stack_metadata "polaris" "${POLARIS_DIR}/docker-compose.yaml" ""
compose_cmd "${POLARIS_DIR}/docker-compose.yaml" "" down --remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
- # polaris-init is a one-shot service. Keep it out of `up --wait`,
- # otherwise Compose reports the successfully exited container as a
- # failed stack startup.
- compose_cmd "${POLARIS_DIR}/docker-compose.yaml" "" up -d --wait
--remove-orphans polaris
- compose_cmd "${POLARIS_DIR}/docker-compose.yaml" "" run --rm --no-deps
polaris-init
+ # polaris-init stays alive after initialization; its healthcheck marks
readiness.
+ compose_cmd "${POLARIS_DIR}/docker-compose.yaml" "" up -d --wait
--remove-orphans
fi
}
@@ -1740,10 +1723,22 @@ start_iceberg_rest() {
export CONTAINER_UID=${CONTAINER_UID}
. "${ROOT}"/docker-compose/iceberg-rest/iceberg-rest_settings.env
envsubst <"${ICEBERG_REST_DIR}/docker-compose.yaml.tpl"
>"${ICEBERG_REST_DIR}/docker-compose.yaml"
+ if [[ "${HOST_OS}" != "Darwin" ]]; then
+ if [[ "${STOP}" -ne 1 ]]; then
+ require_rootless_host_network
+ fi
+ # A host HDFS client must reach the DataNode address returned by the
+ # NameNode. A rootless bridge address is not routable from the host.
+ envsubst <"${ICEBERG_REST_DIR}/hdfs-rootless.yaml.tpl"
>"${ICEBERG_REST_DIR}/hdfs-rootless.yaml"
+ docker_cli compose -f "${ICEBERG_REST_DIR}/docker-compose.yaml" \
+ -f "${ICEBERG_REST_DIR}/hdfs-rootless.yaml" config \
+ >"${ICEBERG_REST_DIR}/docker-compose.rootless.yaml"
+ mv "${ICEBERG_REST_DIR}/docker-compose.rootless.yaml"
"${ICEBERG_REST_DIR}/docker-compose.yaml"
+ fi
register_stack_metadata "iceberg-rest"
"${ICEBERG_REST_DIR}/docker-compose.yaml" ""
compose_cmd "${ICEBERG_REST_DIR}/docker-compose.yaml" "" down
--remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
- # Start all three REST catalogs (S3, OSS, COS)
+ # Start the REST catalogs and their built-in HDFS.
compose_cmd "${ICEBERG_REST_DIR}/docker-compose.yaml" "" up -d
--remove-orphans --wait
fi
}
@@ -1754,7 +1749,7 @@ reserve_ports
# Ensure hive data is downloaded before starting hive2/hive3, but only once
need_prepare_hive_data=0
-if [[ "$NEED_LOAD_DATA" -eq 1 ]]; then
+if [[ "${STOP}" -ne 1 && "$NEED_LOAD_DATA" -eq 1 ]]; then
if [[ "${RUN_HIVE2}" -eq 1 ]] || [[ "${RUN_HIVE3}" -eq 1 ]]; then
if [[ "${HIVE_MODE}" == "refresh" || "${HIVE_MODE}" == "rebuild" ]];
then
need_prepare_hive_data=1
diff --git a/docker/thirdparties/test/docker-command-test.sh
b/docker/thirdparties/test/docker-command-test.sh
new file mode 100755
index 00000000000..3d7c0eab304
--- /dev/null
+++ b/docker/thirdparties/test/docker-command-test.sh
@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+TEST_DIR="$(mktemp -d /tmp/doris-docker-test.XXXXXX)"
+TEST_LOG="${TEST_DIR}/commands.log"
+ORIGINAL_PATH="${PATH}"
+trap 'rm -rf "${TEST_DIR}"' EXIT
+
+fail() {
+ echo "FAIL: $*" >&2
+ exit 1
+}
+
+mkdir -p "${TEST_DIR}/bin"
+
+cat >"${TEST_DIR}/bin/docker" <<'EOF'
+#!/usr/bin/env bash
+if [[ "${1:-}" == "info" ]]; then
+ if [[ "${DOCKER_INFO_EXIT:-0}" -ne 0 ]]; then
+ exit "${DOCKER_INFO_EXIT}"
+ fi
+ if [[ "${2:-}" == "--format" ]]; then
+ if [[ "${3}" == '{{.ServerVersion}}' ]]; then
+ printf '%s\n' "${DOCKER_SERVER_VERSION}"
+ else
+ printf '["name=%s"]\n' "${DOCKER_SECURITY_MODE}"
+ fi
+ fi
+ exit 0
+fi
+printf 'docker %s\n' "$*" >>"${TEST_LOG}"
+EOF
+
+cat >"${TEST_DIR}/bin/sudo" <<'EOF'
+#!/usr/bin/env bash
+printf 'sudo %s\n' "$*" >>"${TEST_LOG}"
+exit 125
+EOF
+
+chmod +x "${TEST_DIR}/bin/docker" "${TEST_DIR}/bin/sudo"
+export PATH="${TEST_DIR}/bin:${ORIGINAL_PATH}"
+export TEST_LOG DOCKER_SECURITY_MODE
+HOST_OS=Linux
+
+# shellcheck source=../docker-command.sh
+. "${ROOT}/docker-command.sh"
+
+docker_cli() {
+ docker "$@"
+}
+
+unset DOCKER_HOST DOCKER_CONTEXT
+DOCKER_SECURITY_MODE=rootless
+configure_docker_access
+: >"${TEST_LOG}"
+export DOCKER_USE_SUDO=1 # A legacy environment must not enable elevation.
+docker_cli ps
+unset DOCKER_USE_SUDO
+grep -Fxq 'docker ps' "${TEST_LOG}" || fail "docker_cli did not call Docker
directly"
+if grep -q '^sudo ' "${TEST_LOG}"; then
+ fail "rootless mode invoked sudo"
+fi
+
+export DOCKER_SERVER_VERSION=29.4.0
+if require_rootless_host_network 2>/dev/null; then
+ fail "rootless Docker before 29.5 was accepted for host networking"
+fi
+DOCKER_SERVER_VERSION=29.5.0
+require_rootless_host_network || fail "rootless Docker 29.5 was rejected"
+DOCKER_SERVER_VERSION=30.0.0
+require_rootless_host_network || fail "newer rootless Docker was rejected"
+
+DOCKER_SECURITY_MODE=seccomp
+: >"${TEST_LOG}"
+if configure_docker_access 2>/dev/null; then
+ fail "Linux accepted a rootful daemon"
+fi
+[[ ! -s "${TEST_LOG}" ]] || fail "a rootful daemon triggered elevation"
+
+# macOS keeps direct Docker Desktop access without Linux rootless detection.
+HOST_OS=Darwin
+configure_docker_access || fail "Docker Desktop was rejected"
+require_rootless_host_network || fail "Linux version requirement applied to
macOS"
+HOST_OS=Linux
+DOCKER_SECURITY_MODE=rootless
+
+# Discover the user's socket even if the default system daemon is accessible.
+export XDG_RUNTIME_DIR="${TEST_DIR}/runtime"
+mkdir -p "${XDG_RUNTIME_DIR}"
+python3 - "${XDG_RUNTIME_DIR}/docker.sock" <<'PY'
+import socket
+import sys
+connection = socket.socket(socket.AF_UNIX)
+connection.bind(sys.argv[1])
+connection.close()
+PY
+unset DOCKER_HOST DOCKER_CONTEXT
+configure_docker_access
+[[ "${DOCKER_HOST}" == "unix://${XDG_RUNTIME_DIR}/docker.sock" ]] || fail
"user socket was not discovered"
+
+DOCKER_HOST=unix:///explicit/docker.sock
+configure_docker_access
+[[ "${DOCKER_HOST}" == unix:///explicit/docker.sock ]] || fail "explicit
endpoint was replaced"
+unset DOCKER_HOST
+export DOCKER_CONTEXT=review-rootless
+configure_docker_access
+[[ -z "${DOCKER_HOST+x}" ]] || fail "explicit context was replaced"
+unset DOCKER_CONTEXT
+
+# A failed daemon probe must not run a host operation or silently escalate it.
+export DOCKER_INFO_EXIT=17
+export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/docker.sock"
+: >"${TEST_LOG}"
+status=0
+(configure_docker_access && touch "${TEST_DIR}/unexpected") 2>/dev/null ||
status=$?
+[[ "${status}" -eq 1 ]] || fail "failed daemon probe did not propagate its
error"
+[[ ! -e "${TEST_DIR}/unexpected" ]] || fail "host command ran after daemon
probe failed"
+[[ ! -s "${TEST_LOG}" ]] || fail "failed daemon probe invoked sudo or another
Docker operation"
+
+echo "PASS"
diff --git a/docker/thirdparties/test/launcher-stop-test.sh
b/docker/thirdparties/test/launcher-stop-test.sh
new file mode 100644
index 00000000000..c8b2b43150b
--- /dev/null
+++ b/docker/thirdparties/test/launcher-stop-test.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Run the real CLI against an isolated fixture tree and a Docker stub.
+# Requires Bash 4+, like the launcher; does not access a real daemon.
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+test_dir="$(mktemp -d /tmp/doris-stop-test.XXXXXX)"
+trap 'rm -rf "${test_dir}"' EXIT
+fixture="${test_dir}/thirdparties"
+mkdir -p "${fixture}" "${test_dir}/bin"
+for file in \
+ run-thirdparties-docker.sh docker-command.sh docker-health.sh \
+ juicefs-helpers.sh custom_settings.env \
+ docker-compose/hive/scripts/bootstrap/bootstrap-groups.sh \
+ docker-compose/hive/hive-3x_settings.env \
+ docker-compose/hive/hadoop-hive.env.tpl \
+ docker-compose/hive/hadoop-hive-3x.env.tpl \
+ docker-compose/hive/hive-3x.yaml.tpl; do
+ mkdir -p "$(dirname "${fixture}/${file}")"
+ cp "${ROOT}/${file}" "${fixture}/${file}"
+done
+# This branch's custom_settings.env sets CONTAINER_UID unconditionally;
rewrite it the way CI does.
+sed 's/^CONTAINER_UID="doris--"/CONTAINER_UID="rootless-stop-test-"/' \
+ "${ROOT}/custom_settings.env" >"${fixture}/custom_settings.env"
+
+export STOP_TEST_DOCKER_LOG="${test_dir}/docker.log"
+export STOP_TEST_PREPARE_MARKER="${test_dir}/prepare-called"
+cat >"${fixture}/docker-compose/hive/scripts/prepare-hive-data.sh" <<'EOF'
+touch "${STOP_TEST_PREPARE_MARKER}"
+exit 99
+EOF
+cat >"${test_dir}/bin/docker" <<'EOF'
+#!/usr/bin/env bash
+if [[ "${1:-}" == info ]]; then
+ echo '["name=rootless"]'
+ exit 0
+fi
+printf '%s\n' "$*" >>"${STOP_TEST_DOCKER_LOG}"
+[[ "${1:-}" == compose ]] || exit 1
+for argument in "$@"; do
+ if [[ "${argument}" == down ]]; then exit 0; fi
+done
+exit 1
+EOF
+cat >"${test_dir}/bin/envsubst" <<'EOF'
+#!/bin/sh
+cat
+EOF
+chmod +x "${test_dir}/bin/docker" "${test_dir}/bin/envsubst"
+
+export PATH="${test_dir}/bin:${PATH}"
+export CONTAINER_UID=rootless-stop-test-
+export DOCKER_HOST=unix:///rootless-stop-test.sock
+export IP_HOST=127.0.0.1
+export HIVE_HOST_ALIAS=localhost
+export JFS_CLUSTER_META=redis://127.0.0.1:6379/0
+export HIVE_MODE=refresh
+bash "${fixture}/run-thirdparties-docker.sh" -c hive3 --stop \
+ >"${test_dir}/launcher.log" 2>&1
+[[ ! -e "${STOP_TEST_PREPARE_MARKER}" ]]
+grep -q 'compose -p rootless-stop-test-hive3 ' "${STOP_TEST_DOCKER_LOG}"
+grep -q ' down$' "${STOP_TEST_DOCKER_LOG}"
+echo "Hive stop bypasses data preparation and only tears down compose"
diff --git a/docker/thirdparties/test/snapshot-hive-baseline-test.sh
b/docker/thirdparties/test/snapshot-hive-baseline-test.sh
new file mode 100644
index 00000000000..8c03a0eee43
--- /dev/null
+++ b/docker/thirdparties/test/snapshot-hive-baseline-test.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Integration test: requires the current user's Docker daemon and Python 3.
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+HOST_OS="$(uname -s)"
+. "${ROOT}/docker-command.sh"
+configure_docker_access
+
+test_dir="$(mktemp -d /tmp/doris-snapshot-test.XXXXXX)"
+prefix="$(basename "${test_dir}" | tr '[:upper:]' '[:lower:]')"
+image="${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}"
+snapshot="${ROOT}/docker-compose/hive/scripts/snapshot-hive-baseline.sh"
+volumes=()
+
+cleanup() {
+ if ((${#volumes[@]})); then
+ docker volume rm "${volumes[@]}" >/dev/null
+ fi
+ rm -rf "${test_dir}"
+}
+trap cleanup EXIT
+
+# Missing volumes must fail before producing an output file.
+if bash "${snapshot}" "${prefix}" "${test_dir}/missing.tar.gz"
>"${test_dir}/missing.log" 2>&1; then
+ echo "FAIL: snapshot accepted missing volumes" >&2
+ exit 1
+fi
+[[ ! -e "${test_dir}/missing.tar.gz" ]]
+
+for suffix in namenode datanode pgdata state; do
+ volumes+=("${prefix}-${suffix}")
+ docker volume create "${prefix}-${suffix}" >/dev/null
+done
+
+docker run --rm --network none --security-opt no-new-privileges \
+ -v "${prefix}-namenode:/fixture/namenode" \
+ -v "${prefix}-datanode:/fixture/datanode" \
+ -v "${prefix}-pgdata:/fixture/pgdata" \
+ -v "${prefix}-state:/fixture/state" \
+ "${image}" sh -ec '
+ for part in namenode datanode pgdata state; do
+ printf "%s" "$part" >"/fixture/$part/payload"
+ done
+ printf legacy >/fixture/state/baseline.version
+ chown 123:456 /fixture/pgdata /fixture/pgdata/payload
+ chmod 700 /fixture/pgdata
+ chmod 600 /fixture/pgdata/payload
+ '
+
+bash "${snapshot}" "${prefix}" "${test_dir}/baseline.tar.gz"
+python3 - "${test_dir}/baseline.tar.gz" <<'PY'
+import sys
+import tarfile
+
+with tarfile.open(sys.argv[1]) as archive:
+ for part in ['namenode', 'datanode', 'pgdata', 'state']:
+ assert archive.extractfile(part + '/payload').read() == part.encode()
+ assert 'state/baseline.version' not in archive.getnames()
+ entry = archive.getmember('pgdata/payload')
+ assert (entry.uid, entry.gid, entry.mode) == (123, 456, 0o600)
+PY
+docker run --rm --network none -v "${prefix}-state:/fixture:ro" \
+ "${image}" test -f /fixture/baseline.version
+
+# An interrupted exporter must preserve an existing baseline and remove its
+# temporary output. Forward info/volume queries to the real rootless daemon.
+printf previous-baseline >"${test_dir}/previous.tar.gz"
+docker() {
+ if [[ "${1:-}" == "run" ]]; then
+ return 42
+ fi
+ command docker "$@"
+}
+export -f docker
+status=0
+bash "${snapshot}" "${prefix}" "${test_dir}/previous.tar.gz" \
+ >"${test_dir}/failure.log" 2>&1 || status=$?
+unset -f docker
+[[ "${status}" -eq 42 ]]
+[[ "$(cat "${test_dir}/previous.tar.gz")" == previous-baseline ]]
+[[ "$(find "${test_dir}" -name 'previous.tar.gz.*' | wc -l | tr -d ' ')" -eq 0
]]
+echo "Snapshot export, UID/GID preservation and failure cleanup passed"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]