Copilot commented on code in PR #8714:
URL: https://github.com/apache/hbase/pull/8714#discussion_r4137462538
##########
dev-support/read-replica/hbase_nightly_read_replica_test.sh:
##########
@@ -15,119 +15,193 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#
-# Run the read-replica Docker integration test suite.
+# hbase_nightly_read_replica_test.sh
+#
+# Outer driver script for the HBase read-replica integration test suite.
+# Builds a dev-support Docker container image (from
+# hbase/dev-support/docker/Dockerfile) using the HBase source tree as
+# build context, then either:
+#
+# - Launches run_read_replica_integration_tests.sh inside that
+# container to execute the full test suite (default), or
+# - Starts a long-lived dev container for interactive exploration
+# (-d|--dev mode).
+#
+# The container uses Docker-outside-of-Docker (DooD) by bind-mounting
+# the host's Docker socket, and mounts .m2 for Maven cache reuse
+# (override the default $HOME location with -m|--m2).
+#
+# For usage information, run: ./hbase_nightly_read_replica_test.sh --help
set -e
REPLICA_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-OUTPUT_DIR="${OUTPUT_DIR:-${REPLICA_DIR}/output}"
export HBASE_ROOT="$(cd "${REPLICA_DIR}/../.." && pwd)"
-export HBASE_IMAGE="hbase-read-replica:${BUILD_NUMBER:-local}"
+DEV_IMAGE_NAME="hbase-dev-support:${BUILD_NUMBER:-local}"
+
+PYTEST_K_VALUE=""
+JAVA_VERSION=""
+M2_DIR="${HOME}"
+DEV_MODE=false
+
+print_usage() {
+ SCRIPT=$(basename "${BASH_SOURCE}")
-KEEP_IMAGE=false
-KEEP_CONTAINERS=false
+ cat << __EOF
+
+hbase_nightly_read_replica_test.sh
+
+Outer driver script for the HBase read-replica integration test suite. Builds a
+dev-support Docker container image and either runs the inner test script
+(run_read_replica_integration_tests.sh) inside it, or starts a long-lived dev
+container for interactive exploration.
+
+Usage: ${SCRIPT} [options]
+
+ -h | --help Show this help message and exit.
+ -d | --dev Start a detached dev container for interactive
+ exploration instead of running the test suite.
+ Prints the container ID and instructions for
+ entering and stopping it.
+ -j | --java-version <ver> JVM version forwarded to the inner test script
+ (run_read_replica_integration_tests.sh), which
+ uses it to set JAVA_HOME inside the container.
+ -k <expression> Pytest -k filter expression forwarded to the
+ inner test script for test selection. (See
+ Pytest documentation on -k for more info.)
+ -m | --m2 <path> Parent directory of the .m2 Maven cache to
+ bind-mount into the container. Defaults to
+ \$HOME. The directory <path>/.m2 will be
+ created if it does not exist.
+
+__EOF
+}
while [[ $# -gt 0 ]]; do
case "$1" in
- -i|--keep-image)
- KEEP_IMAGE=true
- shift
+ -h|--help)
+ print_usage
+ exit 0
;;
- -c|--keep-containers)
- KEEP_CONTAINERS=true
+ -d|--dev)
+ DEV_MODE=true
shift
;;
+ -j|--java-version)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ JAVA_VERSION="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
+ -k)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ PYTEST_K_VALUE="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
+ -m|--m2)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ M2_DIR="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
*)
- echo "Unknown option: $1"
- echo "Usage: $0 [-i|--keep-image] [-c|--keep-containers]"
+ echo "Unknown option: $1" >&2
+ print_usage >&2
exit 1
Review Comment:
The outer driver no longer accepts `-i/--keep-image` or
`-c/--keep-containers`, although the existing README documents those options
and the inner runner still implements them. Existing local debugging commands
now fail in this `Unknown option` branch; either preserve and forward these
options for compatibility or update the documented interface and provide an
equivalent migration path.
##########
dev-support/read-replica/hbase_nightly_read_replica_test.sh:
##########
@@ -15,119 +15,193 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#
-# Run the read-replica Docker integration test suite.
+# hbase_nightly_read_replica_test.sh
+#
+# Outer driver script for the HBase read-replica integration test suite.
+# Builds a dev-support Docker container image (from
+# hbase/dev-support/docker/Dockerfile) using the HBase source tree as
+# build context, then either:
+#
+# - Launches run_read_replica_integration_tests.sh inside that
+# container to execute the full test suite (default), or
+# - Starts a long-lived dev container for interactive exploration
+# (-d|--dev mode).
+#
+# The container uses Docker-outside-of-Docker (DooD) by bind-mounting
+# the host's Docker socket, and mounts .m2 for Maven cache reuse
+# (override the default $HOME location with -m|--m2).
+#
+# For usage information, run: ./hbase_nightly_read_replica_test.sh --help
set -e
REPLICA_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-OUTPUT_DIR="${OUTPUT_DIR:-${REPLICA_DIR}/output}"
export HBASE_ROOT="$(cd "${REPLICA_DIR}/../.." && pwd)"
-export HBASE_IMAGE="hbase-read-replica:${BUILD_NUMBER:-local}"
+DEV_IMAGE_NAME="hbase-dev-support:${BUILD_NUMBER:-local}"
+
+PYTEST_K_VALUE=""
+JAVA_VERSION=""
+M2_DIR="${HOME}"
+DEV_MODE=false
+
+print_usage() {
+ SCRIPT=$(basename "${BASH_SOURCE}")
-KEEP_IMAGE=false
-KEEP_CONTAINERS=false
+ cat << __EOF
+
+hbase_nightly_read_replica_test.sh
+
+Outer driver script for the HBase read-replica integration test suite. Builds a
+dev-support Docker container image and either runs the inner test script
+(run_read_replica_integration_tests.sh) inside it, or starts a long-lived dev
+container for interactive exploration.
+
+Usage: ${SCRIPT} [options]
+
+ -h | --help Show this help message and exit.
+ -d | --dev Start a detached dev container for interactive
+ exploration instead of running the test suite.
+ Prints the container ID and instructions for
+ entering and stopping it.
+ -j | --java-version <ver> JVM version forwarded to the inner test script
+ (run_read_replica_integration_tests.sh), which
+ uses it to set JAVA_HOME inside the container.
+ -k <expression> Pytest -k filter expression forwarded to the
+ inner test script for test selection. (See
+ Pytest documentation on -k for more info.)
+ -m | --m2 <path> Parent directory of the .m2 Maven cache to
+ bind-mount into the container. Defaults to
+ \$HOME. The directory <path>/.m2 will be
+ created if it does not exist.
+
+__EOF
+}
while [[ $# -gt 0 ]]; do
case "$1" in
- -i|--keep-image)
- KEEP_IMAGE=true
- shift
+ -h|--help)
+ print_usage
+ exit 0
;;
- -c|--keep-containers)
- KEEP_CONTAINERS=true
+ -d|--dev)
+ DEV_MODE=true
shift
;;
+ -j|--java-version)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ JAVA_VERSION="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
+ -k)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ PYTEST_K_VALUE="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
+ -m|--m2)
+ if [[ -n "$2" && "$2" != -* ]]; then
+ M2_DIR="$2"
+ shift 2
+ else
+ echo "Error: Argument for $1 is missing" >&2
+ print_usage >&2
+ exit 1
+ fi
+ ;;
*)
- echo "Unknown option: $1"
- echo "Usage: $0 [-i|--keep-image] [-c|--keep-containers]"
+ echo "Unknown option: $1" >&2
+ print_usage >&2
exit 1
;;
esac
done
-echo "Replica dir: ${REPLICA_DIR}"
-echo "Output dir: ${OUTPUT_DIR}"
-echo "HBase root: ${HBASE_ROOT}"
-
-echo "Changing to replica dir: ${REPLICA_DIR}"
-cd "${REPLICA_DIR}"
+echo "=== HBase Read-Replica Integration Test Driver ==="
+echo "HBase Root: ${HBASE_ROOT}"
+echo "Replica Dir: ${REPLICA_DIR}"
+echo "Dev Container Image: ${DEV_IMAGE_NAME}"
+echo "M2 Dir: ${M2_DIR}/.m2"
-echo "Sourcing environment file: $(pwd)/.env"
-set -a
-source .env
-set +a
+# Build the dev-support container image using HBASE_ROOT as the build context
+echo "Building dev-support Docker image..."
+docker build --platform linux/amd64 \
+ -t "${DEV_IMAGE_NAME}" \
+ -f "${HBASE_ROOT}/dev-support/docker/Dockerfile" \
+ "${HBASE_ROOT}"
-echo "HBASE_IMAGE=${HBASE_IMAGE}"
-echo "ACTIVE_CLUSTER_CONF_DIR=${ACTIVE_CLUSTER_CONF_DIR}"
-echo "REPLICA_CLUSTER_CONF_DIR=${REPLICA_CLUSTER_CONF_DIR}"
-echo "DOCKER_COMPOSE_FILE=${DOCKER_COMPOSE_FILE}"
-echo "HBASE_DATA_STORE_ROOT=${HBASE_DATA_STORE_ROOT}"
-echo "realpath of HBASE_DATA_STORE_ROOT=$(realpath ${HBASE_DATA_STORE_ROOT})"
-
-# Clone HBase source for Docker build context (Docker COPY doesn't follow
symlinks)
-echo "Cloning HBase source into ${REPLICA_DIR}/hbase for Docker build
context..."
-rm -rf "${REPLICA_DIR}/hbase"
-git clone --local "${HBASE_ROOT}" "${REPLICA_DIR}/hbase"
-rm -rf "${REPLICA_DIR}/hbase/.git"
-
-cleanup() {
+cleanup_host() {
local exit_code=$?
- if [ ${exit_code} -ne 0 ]; then
- echo "=== FAILURE ==="
- echo "An error occurred during this stage in the Jenkins run."
- fi
- if [ "${KEEP_CONTAINERS}" = "false" ]; then
- echo "=== Cleanup: Stopping Docker containers ==="
- docker compose -f "${DOCKER_COMPOSE_FILE}" down 2>/dev/null || true
+ if [ -z "${BUILD_NUMBER}" ]; then
+ echo "Local execution complete. Preserving local container image
${DEV_IMAGE_NAME}."
else
- echo "=== Cleanup: Keeping Docker containers (--keep-containers) ==="
+ echo "Jenkins execution complete. Cleaning up image ${DEV_IMAGE_NAME}..."
+ docker rmi --force "${DEV_IMAGE_NAME}" 2>/dev/null || true
fi
- if [ "${KEEP_IMAGE}" = "false" ]; then
- echo "=== Cleanup: Removing Docker image: ${HBASE_IMAGE} ==="
- docker rmi --force "${HBASE_IMAGE}" 2>/dev/null || true
- else
- echo "=== Cleanup: Keeping Docker image: ${HBASE_IMAGE} (--keep-image) ==="
- fi
- echo "=== Cleanup: Deleting cloned HBase directory: ${REPLICA_DIR}/hbase ==="
- rm -rf "${REPLICA_DIR}/hbase"
exit "${exit_code}"
}
-trap cleanup EXIT
-
-# Copy latest proto file from source
-echo "Copying latest version of ActiveClusterSuffix.proto to
$(pwd)/python/proto/"
-cp
"${HBASE_ROOT}/hbase-protocol-shaded/src/main/protobuf/server/ActiveClusterSuffix.proto"
\
- python/proto/
-
-export PYTHONPATH="$(pwd)"
-echo "Set PYTHONPATH=${PYTHONPATH}"
-
-# Create Python environment
-echo "Creating Python environment: .venv"
-python3 -m venv .venv
-source .venv/bin/activate
-
-# Install Python dependencies
-echo "Installing Python libraries"
-pip install --upgrade pip
-pip install -r requirements.txt
-
-# Compile protobuf
-echo "Compiling Protobuf"
-python3 python/proto/proto_compiler.py
-
-# Build Docker images
-echo "Building hbase-docker image"
-./build-images.sh
-
-# Run read-replica integration test suite
-echo "Starting read-replica integration test scripts"
-echo "Starting read-replica integration test suite via Pytest..."
-pytest --html="${OUTPUT_DIR}/read-replica-nightly-test-report.html" \
- --self-contained-html \
- --junitxml="${OUTPUT_DIR}/read-replica-nightly-test-results.xml" \
- python/test/test_read_replica_feature.py
-
-echo "=== Success: All read-replica integration tests passed. ==="
+trap cleanup_host EXIT
+
+# Ensure host .m2 directory exists for caching
+mkdir -p "${M2_DIR}/.m2"
+
+if [ "${DEV_MODE}" = "true" ]; then
+ # Start a detached dev container for interactive exploration
+ echo "Starting dev container in background..."
+ CONTAINER_ID=$(docker run -d \
+ --platform linux/amd64 \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ -v "${HBASE_ROOT}:${HBASE_ROOT}" \
+ -v "${M2_DIR}/.m2:/root/.m2" \
+ -e OUTPUT_DIR="${OUTPUT_DIR}" \
+ -e BUILD_NUMBER="${BUILD_NUMBER:-local}" \
+ -w "${REPLICA_DIR}" \
+ "${DEV_IMAGE_NAME}" \
+ sleep infinity)
+
+ echo ""
+ echo "=== Dev container is ready ==="
+ echo "Container ID: ${CONTAINER_ID}"
+ echo "Image: ${DEV_IMAGE_NAME}"
+ echo ""
+ echo "Enter the container:"
+ echo ""
+ echo "docker exec -it ${CONTAINER_ID} bash"
+ echo ""
+ echo "Stop and remove the container when done:"
+ echo ""
+ echo "docker stop ${CONTAINER_ID} && docker rm ${CONTAINER_ID}"
+ echo ""
+else
+ JAVA_VERSION_ARGS=()
+ if [[ -n "${JAVA_VERSION}" ]]; then
+ JAVA_VERSION_ARGS=(-j "${JAVA_VERSION}")
+ fi
+
+ PYTEST_K_ARGS=()
+ if [[ -n "${PYTEST_K_VALUE}" ]]; then
+ PYTEST_K_ARGS=(-k "${PYTEST_K_VALUE}")
+ fi
+
+ # Run the inner test script inside the dev-support container via DooD
+ echo "Launching dev container and starting test suite..."
+ docker run --rm \
+ --platform linux/amd64 \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ -v "${HBASE_ROOT}:${HBASE_ROOT}" \
+ -v "${M2_DIR}/.m2:/root/.m2" \
+ -e OUTPUT_DIR="${OUTPUT_DIR}" \
+ -e BUILD_NUMBER="${BUILD_NUMBER:-local}" \
+ -w "${REPLICA_DIR}" \
Review Comment:
On Jenkins, `OUTPUT_DIR` is an absolute workspace path outside `HBASE_ROOT`,
but this container only bind-mounts `HBASE_ROOT` and `.m2`. The inner pytest
process therefore writes the HTML/JUnit files into the disposable container
filesystem, so the Jenkins post stage cannot find or publish the results.
Bind-mount the effective output directory (and default it under `REPLICA_DIR`
for local runs) before invoking the inner script.
##########
dev-support/read-replica/run_read_replica_integration_tests.sh:
##########
@@ -0,0 +1,215 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# run_read_replica_integration_tests.sh
+#
+# Inner test runner executed INSIDE the Docker container built from
+# hbase/dev-support/docker/Dockerfile. In a typical test run, this
+# script is invoked by hbase_nightly_read_replica_test.sh. It can be
+# run on its own as well as long as it is done within a the container
+# mentioned above.
Review Comment:
This comment contains the duplicated article "a the", which makes the usage
guidance grammatically incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
