bitflicker64 commented on code in PR #3149: URL: https://github.com/apache/hugegraph/pull/3149#discussion_r3852385993
########## docker/docker-compose-hubble.yml: ########## @@ -0,0 +1,64 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Hubble add-on for the distributed cluster defined in +# docker-compose-3pd-3store-3server.yml. Requires the pre-created cluster +# network (docker network create hugegraph-net). See "Hubble for the +# 3-Node Cluster" in docker/README.md for the attach and combined flows. + +networks: + hg-net: + external: true + name: ${HUGEGRAPH_NETWORK:-hugegraph-net} + +volumes: + hg-hubble-db: + # Explicit names so attach (-p hugegraph-hubble) and combined + # (project hugegraph-3x3) share the same physical volumes. + name: ${HUBBLE_DB_VOLUME:-hugegraph-hubble-db} Review Comment: These volumes are shared across both flows by fixed name but are not `external`, while the equally shared network is. That asymmetry is the bug: `down -v` from either project removes them. I checked this against Compose directly. A fixed-name non-external volume is removed by `down -v` even from a different project that merely declares the same name, and an `external: true` volume survives. So leaving the attach flow with `down -v` silently destroys the combined flow's H2 database and uploads, which contradicts "Recreating the container does not discard that state" in the README. Either declare them `external` (pre-created like the network) or warn about `down -v` explicitly. The current combination gives neither protection nor warning. ########## docker/README.md: ########## @@ -168,14 +175,142 @@ To validate local images without Compose replacing them with remote `latest`: ## 3-Node Cluster Quickstart +The cluster and the Hubble add-on share one named Docker network so Hubble +can attach to a running cluster without touching it. Treat that network as a +trust boundary: only the Server layer performs real authentication. Store +serves its control APIs with no credentials at all, and while PD's REST +control APIs do require an `Authorization` header, PD only checks that the +Basic-auth *user* is one of its internal service names and never validates +the password — so any client on the network can read and drive them. Note +that this stack depends on that behaviour: Hubble reaches PD as the service +name `hubble` with an empty password, so tightening PD's credential check +would also break Hubble's operations view. Because of that, PD and Store +control-plane ports and the three Server REST ports bind to `127.0.0.1` by +default, so an unauthenticated control plane is not reachable from outside the +host. Publish them more widely only by setting `HUGEGRAPH_CONTROL_PLANE_HOST` +or `HUGEGRAPH_SERVER_PUBLISH_HOST`, and only behind a network ACL or TLS +terminator you control. Any container on the host can still join the network +by declaring the well-known name. One-time setup: +write the required credentials to a mode-600 `docker/.env` and create the +network. +The cluster file requires both credentials — the admin password enables +authentication, and every Server replica must share one token secret so a +token issued by any server validates on all of them. The `:?` guards fire +on every Compose subcommand, including `down`. + +```bash +( + set -eu + cd docker + command -v openssl >/dev/null 2>&1 || { echo "openssl not found" >&2; exit 1; } + [ -e .env ] || install -m 600 /dev/null .env + chmod 600 .env + # Keep appends on their own lines even if the file was hand-edited. + [ ! -s .env ] || [ -z "$(tail -c1 .env)" ] || printf '\n' >> .env + pat='^[[:space:]]*(export[[:space:]]+)?' + if ! grep -Eq "${pat}HUGEGRAPH_ADMIN_PASSWORD=" .env; then + admin_password="$(openssl rand -base64 12)" + printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\n" "${admin_password}" >> .env + unset admin_password + fi + if ! grep -Eq "${pat}HUGEGRAPH_AUTH_TOKEN_SECRET=" .env; then + token_secret="$(openssl rand -hex 32)" + printf "HUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" "${token_secret}" >> .env + unset token_secret + fi + # Parse values as data. Do not source .env as shell: a value such as + # HUGEGRAPH_ADMIN_PASSWORD=$(...) would run on the operator host. + # The optional export group is capture 1; the quoted value is capture 2. + read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env + } + if ! admin_value="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)"; then + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 + fi + if ! token_value="$(read_dotenv_value HUGEGRAPH_AUTH_TOKEN_SECRET)"; then + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must use the documented single-quoted format" >&2 + exit 1 + fi + [ -n "${admin_value}" ] || { + echo "HUGEGRAPH_ADMIN_PASSWORD must be non-empty in docker/.env" >&2 + exit 1 + } + [ "${#token_value}" -ge 32 ] || { + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must be at least 32 bytes in docker/.env" >&2 + exit 1 + } + unset admin_value token_value + # The shared cluster network. To override the name, export + # HUGEGRAPH_NETWORK in this shell before running the block — a value + # in docker/.env is read by Compose, not by this script. + net="${HUGEGRAPH_NETWORK:-hugegraph-net}" + docker network inspect "${net}" >/dev/null 2>&1 || + docker network create "${net}" + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + docker compose -f docker-compose-3pd-3store-3server.yml config --quiet +) +``` + +Then start the cluster: + ```bash cd docker -HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose-3pd-3store-3server.yml up -d +docker compose -f docker-compose-3pd-3store-3server.yml up -d -# To stop and remove all data volumes (clean restart) +# To stop and remove all data volumes (clean restart). +# The external hugegraph-net network is intentionally left in place. +# If the Hubble add-on is running, see "Hubble for the 3-Node Cluster" +# for the teardown that matches how it was started. docker compose -f docker-compose-3pd-3store-3server.yml down -v ``` +Pin a release by setting `HUGEGRAPH_VERSION` in `docker/.env` — the +cluster, the Hubble add-on, and the single-node quickstart file all read +it from there, so those versions cannot drift apart +(`docker-compose.dev.yml` builds PD/Store/Server from source and defaults +Hubble to `hugegraph/hubble:latest`; set `HUBBLE_IMAGE` to pin it). +Unpinned, the images default to `latest`; note the authenticated PD/Hubble +integration requires a release newer than `1.7.x`. On an older image the +Server ignores `PASSWORD` and `HG_SERVER_AUTH_TOKEN_SECRET`, which would +leave you an unauthenticated cluster. Two defaults keep that from passing +unnoticed: the cluster and add-on use `pull_policy: always`, so a stale +cached `latest` is refreshed on every `up -d`, and the Server healthcheck +requires an unauthenticated graph request to return `401` and an +authenticated one to return `200`. An image that does not enforce +authentication therefore never reports healthy, and `up -d --wait` fails +instead of handing you a false green. + +Each Server registers its `HG_SERVER_REST_URL` with PD. The defaults +(`server0`, `server1`, `server2`) are Docker-network names, which is correct +for Hubble and anything else attached to `hugegraph-net`, but they do not +resolve from an unrelated host network. A PD-aware client running outside +Docker will receive names it cannot reach. For that case, set +`HUGEGRAPH_SERVER0_REST_URL`, `HUGEGRAPH_SERVER1_REST_URL`, and Review Comment: This override cannot produce a working setup for server1 and server2. `HG_SERVER_REST_URL` becomes `restserver.url`, which is the address Grizzly binds to (`RestServer.java`, `GrizzlyHttpServerFactory.createHttpServer(uri, ...)`), not only the address registered with PD. So an address that resolves from an external client is either not local to the container (BindException, crash loop) or, with split-horizon DNS, makes the Server listen on a port the published mapping does not forward, since the mapping keeps container port 8080 hard-coded (`:8081:8080`). The healthcheck then probes the same URL from inside the container, so that second case reports healthy behind a dead published port. The separation you need already exists in the code: `GraphManager.loadServices` uses `server.urls_to_pd` instead of `restserver.url`, but only when `server.deploy_in_k8s` is true. Ungating that is the real fix. Until then this paragraph promises something the stack cannot do. Same applies to `docker-compose-3x3.non-auth.yml:56`, which probes `$${HG_SERVER_REST_URL}/versions`. ########## docker/docker-compose-3pd-3store-3server.yml: ########## @@ -58,21 +65,39 @@ x-store-common: &store-common retries: 40 start_period: 120s +# Shared Server environment; each server node adds its own +# HG_SERVER_REST_URL on top of this map. +x-server-env: &server-env + STORE_REST: store0:8520 + HG_SERVER_BACKEND: hstore + HG_SERVER_PD_PEERS: pd0:8686,pd1:8686,pd2:8686 + # Register every Server replica with PD under one application name so + # PD-aware clients such as Hubble discover the cluster as one logical + # Server; `hg` matches the Hubble configuration. + HG_SERVER_CLUSTER: hg + HG_SERVER_USE_PD: "true" + HG_SERVER_MIN_FREE_MEMORY: "0" + HG_SERVER_INIT_STORE_ENABLED: "false" + # All replicas must share one token secret so a token issued by any + # server validates on every other server. + HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:?Set a shared auth token secret} + PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:?Set a non-default admin password} + x-server-common: &server-common image: hugegraph/server:${HUGEGRAPH_VERSION:-latest} - pull_policy: missing + pull_policy: always restart: unless-stopped networks: [hg-net] depends_on: store0: { condition: service_healthy } store1: { condition: service_healthy } store2: { condition: service_healthy } - environment: - STORE_REST: store0:8520 - HG_SERVER_BACKEND: hstore - HG_SERVER_PD_PEERS: pd0:8686,pd1:8686,pd2:8686 healthcheck: - test: ["CMD-SHELL", "curl -fsS http://localhost:8080/versions >/dev/null || exit 1"] + # Prove the image actually enforces authentication. Older cached images + # may accept PASSWORD but ignore it while still returning healthy on + # /versions, so readiness requires both 401 without credentials and 200 + # with the configured admin password. + test: ["CMD-SHELL", "base=$${HG_SERVER_REST_URL}; unauth=$$(curl -s -o /dev/null -w '%{http_code}' \"$${base}/graphs/hugegraph/schema/vertexlabels\"); auth=$$(curl -s -o /dev/null -w '%{http_code}' -u \"admin:$${PASSWORD}\" \"$${base}/graphs/hugegraph/schema/vertexlabels\"); test \"$${unauth}\" = 401 && test \"$${auth}\" = 200"] Review Comment: Two problems with this healthcheck, both stemming from running an authenticated probe every 10 seconds forever. 1. It ties liveness to the *initial* admin password. `PASSWORD` only seeds the admin on first init (`GraphManager.initAdminUserIfNeeded` skips an existing admin, and the variable table below says the same). Once an operator rotates the admin password through the API, the authenticated probe returns 401 and all three Servers go permanently unhealthy on a cluster that is working fine. Any later `up -d --wait` fails, and so does any health-gated `depends_on` a user adds. It also assumes the `hugegraph` graph exists for the lifetime of the deployment. 2. It puts the admin password in process argv every 10 seconds in all three Server containers, where `ps` on the host or `/proc/*/cmdline` in the container exposes it. The README's own verification snippet refuses to do exactly this and says why: passing the credential through `--config` "keeps it out of argv, where `ps` would expose it". Piping a config into `curl -K -` instead of using `-u` addresses the second point, and keeps the credential out of argv because `printf` is a builtin in the image's `/bin/sh`. For the first, proving the image enforces auth is a one-time property: a startup gate, or the smoke test this PR already adds, fits it better than a permanent probe. ########## docker/README.md: ########## @@ -196,11 +331,268 @@ curl http://localhost:8520/v1/health # Check Server (Graph API) curl http://localhost:8080/versions -# List registered stores via PD -curl http://localhost:8620/v1/stores +# Every PD endpoint except /v1/health, /actuator/* and /v1/prom/targets/* +# needs an Authorization header. PD only checks that the Basic-auth user is +# one of its internal service names, so the empty password below is enough +# — and it grants the full PD control plane, writes included, not just these +# reads. That is exactly why the shared network must be treated as a trust +# boundary. Without the header PD answers with an exception body, not data. +pd_auth="Authorization: Basic $(printf 'hubble:' | base64)" + +# List registered stores via PD (expect three, each "state":"Up") +curl -H "${pd_auth}" http://localhost:8620/v1/stores # List partitions -curl http://localhost:8620/v1/partitions +curl -H "${pd_auth}" http://localhost:8620/v1/partitions +``` + +Confirm authentication actually engaged — `/versions` stays open by design, +so it cannot tell you whether auth is on. A graph read without credentials +must be rejected: + +```bash +cd docker +# Expect 401 on all three replicas +for port in 8080 8081 8082; do + curl -s -o /dev/null -w "${port}: %{http_code}\n" \ + "http://localhost:${port}/graphs/hugegraph/schema/vertexlabels"; +done + +# And a signed-in read must succeed. Parse the generated single-quoted +# value without executing docker/.env as shell. Passing the credential +# through --config keeps it out of argv, where `ps` would expose it. +read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env +} +HUGEGRAPH_ADMIN_PASSWORD="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)" || { + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 +} +curl -s -o /dev/null -w '%{http_code}\n' \ + --config <(printf 'user = "admin:%s"\n' "${HUGEGRAPH_ADMIN_PASSWORD}") \ + http://localhost:8080/graphs/hugegraph/schema/vertexlabels +unset HUGEGRAPH_ADMIN_PASSWORD +``` + +`200` from the second command with `401` from all three of the first means +authentication is on and working. If the first command returns `200`, the +running image ignored `PASSWORD` and the cluster is **unauthenticated** — +the most likely cause is an image older than the release this integration +needs (see the version note in the quickstart above). + +### Running without authentication + +The cluster above authenticates, and that is the right default: it is the +configuration most people copy, and an open graph database on a reachable +port is a bad surprise. For a throwaway local trial on a trusted network you +can opt out with `docker-compose-3x3.non-auth.yml`, which drops the admin +password and the JWT secret from all three Servers. Anyone who can reach the +published ports then has full read and write access to every graph, so do not +use it anywhere else. + +```bash +cd docker + +# No docker/.env and no credential setup are needed for this mode. The two Review Comment: Two gaps make this mode unusable as written, despite "no credential setup are needed": 1. The external network is never created in this flow. The only place that creates it before you need it is the credentialed setup block this line tells the reader to skip, so the documented `up -d` fails on a fresh host with `network hugegraph-net declared as external, but could not be found`. The troubleshooting entry covers it, but only after the failure. 2. No teardown is documented, and the base file's `:?` guards fire on every subcommand, `down` included. A later `down` in a fresh shell therefore demands credentials this mode says it does not need, and the section explicitly forbids the obvious workaround of putting them in `.env`. Adding the `docker network create` line and a teardown command carrying the same placeholders would close both. ########## docker/README.md: ########## @@ -196,11 +331,268 @@ curl http://localhost:8520/v1/health # Check Server (Graph API) curl http://localhost:8080/versions -# List registered stores via PD -curl http://localhost:8620/v1/stores +# Every PD endpoint except /v1/health, /actuator/* and /v1/prom/targets/* +# needs an Authorization header. PD only checks that the Basic-auth user is +# one of its internal service names, so the empty password below is enough +# — and it grants the full PD control plane, writes included, not just these +# reads. That is exactly why the shared network must be treated as a trust +# boundary. Without the header PD answers with an exception body, not data. +pd_auth="Authorization: Basic $(printf 'hubble:' | base64)" + +# List registered stores via PD (expect three, each "state":"Up") +curl -H "${pd_auth}" http://localhost:8620/v1/stores # List partitions -curl http://localhost:8620/v1/partitions +curl -H "${pd_auth}" http://localhost:8620/v1/partitions +``` + +Confirm authentication actually engaged — `/versions` stays open by design, +so it cannot tell you whether auth is on. A graph read without credentials +must be rejected: + +```bash +cd docker +# Expect 401 on all three replicas +for port in 8080 8081 8082; do + curl -s -o /dev/null -w "${port}: %{http_code}\n" \ + "http://localhost:${port}/graphs/hugegraph/schema/vertexlabels"; +done + +# And a signed-in read must succeed. Parse the generated single-quoted +# value without executing docker/.env as shell. Passing the credential +# through --config keeps it out of argv, where `ps` would expose it. +read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env +} +HUGEGRAPH_ADMIN_PASSWORD="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)" || { Review Comment: This guard does not fire in the case it is written for. When the key is present but not single-quoted, `grep -Ec` still counts 1 and `sed` prints nothing while exiting 0, so `read_dotenv_value` succeeds with an empty value and the `||` branch never runs. I reproduced this in bash, zsh and dash: the block then curls with `admin:` and an empty password, gets 401, and the interpretation text below sends the operator off to blame the image. The setup block avoids this only through its separate non-empty check, whose message is itself misleading (it reports "must be non-empty" for a value that is non-empty but unquoted). Making `read_dotenv_value` fail when `sed` produces nothing fixes both blocks. ########## docker/README.md: ########## @@ -196,11 +331,268 @@ curl http://localhost:8520/v1/health # Check Server (Graph API) curl http://localhost:8080/versions -# List registered stores via PD -curl http://localhost:8620/v1/stores +# Every PD endpoint except /v1/health, /actuator/* and /v1/prom/targets/* +# needs an Authorization header. PD only checks that the Basic-auth user is +# one of its internal service names, so the empty password below is enough +# — and it grants the full PD control plane, writes included, not just these +# reads. That is exactly why the shared network must be treated as a trust +# boundary. Without the header PD answers with an exception body, not data. +pd_auth="Authorization: Basic $(printf 'hubble:' | base64)" + +# List registered stores via PD (expect three, each "state":"Up") +curl -H "${pd_auth}" http://localhost:8620/v1/stores # List partitions -curl http://localhost:8620/v1/partitions +curl -H "${pd_auth}" http://localhost:8620/v1/partitions +``` + +Confirm authentication actually engaged — `/versions` stays open by design, +so it cannot tell you whether auth is on. A graph read without credentials +must be rejected: + +```bash +cd docker Review Comment: Unlike the setup block, this one is not wrapped in a subshell but still calls `exit 1` below. Pasting it into an interactive shell with a malformed `.env` (missing or duplicated key) terminates the operator's session. I reproduced this in bash and zsh, interactive and non-interactive; wrapping the same lines in `( ... )` as the setup block does fixes it. ########## docker/docker-compose-3pd-3store-3server.yml: ########## @@ -31,8 +36,10 @@ volumes: # ── Shared service defaults ────────────────────────────────────────── x-pd-common: &pd-common + # Pin a release via HUGEGRAPH_VERSION in docker/.env; unset, the image + # tags default to latest. All Compose files here read the same variable. image: hugegraph/pd:${HUGEGRAPH_VERSION:-latest} - pull_policy: missing + pull_policy: always Review Comment: `pull_policy: always` here has no override variable, unlike the add-on's `HUBBLE_PULL_POLICY` and the single-node file's `HUGEGRAPH_SERVER_PULL_POLICY`. That makes the cluster unusable offline: I confirmed against an unreachable registry that `up -d` exits non-zero and starts nothing even when every image is already cached, because the pull error is only tolerated for services that have a `build:` section. It also silently replaces a locally built tag. ```suggestion pull_policy: ${HUGEGRAPH_PULL_POLICY:-always} ``` The same edit is needed on the store and server anchors (lines 54 and 88), or the offline case still fails. The default stays `always`, which is what keeps a stale cached `latest` from satisfying an auth-sensitive deployment; this only adds a deliberate opt-out. ########## .github/workflows/server-ci.yml: ########## @@ -138,6 +140,322 @@ jobs: check_compose docker/docker-compose.yml always always check_compose docker/docker-compose.dev.yml build missing + check_cluster_compose() { + local cluster="docker/docker-compose-3pd-3store-3server.yml" + local addon="docker/docker-compose-hubble.yml" + local rendered + local token_fixture=ci-test-token-secret-32-bytes-long + rendered="$(mktemp)" + if [ "${#token_fixture}" -lt 32 ]; then + echo "CI token fixture must be at least 32 bytes" >&2 + return 1 + fi + + # RETURN only: an EXIT trap would fire after this function's + # `local rendered` has gone out of scope, which `set -u` turns + # into an "unbound variable" error. A hard errexit abort can + # therefore still leak one temp file, which is acceptable on an + # ephemeral runner. + trap 'rm -f "$rendered"' RETURN + + # --env-file /dev/null on every invocation: Compose otherwise reads + # docker/.env automatically, and the quickstart tells operators to + # create one holding exactly the credentials these assertions + # control. Without it the checks pass in CI (which has no .env) but + # report false failures for anyone running them locally after + # following the quickstart, and the add-on render below — which + # deliberately unsets the variables to assert their defaults — would + # read whatever HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION that .env holds. + # Pinning an empty env file makes both renders depend only on what + # each invocation sets explicitly. + + # Both cluster credentials are required and may not be empty. + # Each case asserts the guard fired for the *intended* variable: + # a bare non-zero exit would also be produced by a YAML error, a + # renamed file, or a missing docker binary. + assert_guard() { # assert_guard <blamed-var> <description> + local var="$1" desc="$2" err + if err="$(docker compose --env-file /dev/null -f "$cluster" \ + config -q 2>&1)"; then + echo "$cluster accepted $desc" >&2 + return 1 + fi + case "$err" in + *"$var"*) : ;; + *) echo "$cluster rejected $desc, but not because of $var: $err" >&2 + return 1 ;; + esac + } + + ( unset HUGEGRAPH_ADMIN_PASSWORD + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an unset admin password" ) + ( export HUGEGRAPH_ADMIN_PASSWORD= + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an empty admin password" ) + ( unset HUGEGRAPH_AUTH_TOKEN_SECRET + export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an unset token secret" ) + ( export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + export HUGEGRAPH_AUTH_TOKEN_SECRET= + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an empty token secret" ) + # The add-on alone must define Hubble and nothing else, join the + # shared external network with its default name, and need no + # credentials or overrides. + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + -u HUGEGRAPH_NETWORK -u HUGEGRAPH_VERSION \ + -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ + -u HUBBLE_DB_VOLUME -u HUBBLE_UPLOAD_VOLUME \ + docker compose --env-file /dev/null -f "$addon" config --format json > "$rendered" + jq -e ' + (.services | keys) == ["hubble"] and + .networks."hg-net".external == true and + .networks."hg-net".name == "hugegraph-net" and + (.services.hubble.networks | has("hg-net")) and + (.services.hubble | has("depends_on") | not) and + .volumes."hg-hubble-db".name == "hugegraph-hubble-db" and + .volumes."hg-hubble-upload-files".name == + "hugegraph-hubble-upload-files" and + .services.hubble.environment.SPRING_DATASOURCE_URL == + "jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE" and + any(.services.hubble.volumes[]; + .target == "/hubble/conf/hugegraph-hubble.properties" and + (.source | endswith("hugegraph-hubble-3x3.properties"))) + and any(.services.hubble.volumes[]; + .source == "hg-hubble-db" and + .target == "/hubble/db") + and any(.services.hubble.volumes[]; + .source == "hg-hubble-upload-files" and + .target == "/hubble/upload-files") + ' "$rendered" >/dev/null + + # The cluster's own default network name must match the add-on's, + # or the attach flow and the cluster land on different networks. + # The combined render below pins an override, so it cannot catch a + # drifting default. + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + env -u HUGEGRAPH_NETWORK -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY \ + -u HUBBLE_PUBLISH_HOST \ + docker compose --env-file /dev/null -f "$cluster" \ + config --format json > "$rendered" + jq -e '.networks."hg-net".name == "hugegraph-net"' \ + "$rendered" >/dev/null + + # The combined render carries the PD-registration and auth + # settings on every server replica and keeps Hubble on loopback. + # Rendered with non-default HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION so + # CI fails if any file stops honoring the overrides (the add-on + # render above covers the defaults). + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + HUGEGRAPH_NETWORK=ci-test-net \ + HUGEGRAPH_VERSION=ci-test-tag \ + env -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ Review Comment: These renders assert defaults for variables they never unset, which reintroduces exactly the local false-failure the comment above says the design prevents. `--env-file /dev/null` stops Compose reading `docker/.env`, but an exported shell variable still wins. Confirmed by rendering the real files: with `HUGEGRAPH_SERVER_PUBLISH_HOST=0.0.0.0` exported, the `host_ip == "127.0.0.1"` assertion fails; same for `HUGEGRAPH_CONTROL_PLANE_HOST`, for `HUGEGRAPH_SERVERn_REST_URL` against the `http://serverN:8080` assertions, and for `HUBBLE_PROPERTIES` against the add-on render's properties-path assertion. These are all variables the README tells operators to export. Adding them to the two `env -u` lists is enough. ########## docker/README.md: ########## @@ -168,14 +175,142 @@ To validate local images without Compose replacing them with remote `latest`: ## 3-Node Cluster Quickstart +The cluster and the Hubble add-on share one named Docker network so Hubble +can attach to a running cluster without touching it. Treat that network as a +trust boundary: only the Server layer performs real authentication. Store +serves its control APIs with no credentials at all, and while PD's REST +control APIs do require an `Authorization` header, PD only checks that the +Basic-auth *user* is one of its internal service names and never validates +the password — so any client on the network can read and drive them. Note +that this stack depends on that behaviour: Hubble reaches PD as the service +name `hubble` with an empty password, so tightening PD's credential check +would also break Hubble's operations view. Because of that, PD and Store +control-plane ports and the three Server REST ports bind to `127.0.0.1` by +default, so an unauthenticated control plane is not reachable from outside the +host. Publish them more widely only by setting `HUGEGRAPH_CONTROL_PLANE_HOST` +or `HUGEGRAPH_SERVER_PUBLISH_HOST`, and only behind a network ACL or TLS +terminator you control. Any container on the host can still join the network +by declaring the well-known name. One-time setup: +write the required credentials to a mode-600 `docker/.env` and create the +network. +The cluster file requires both credentials — the admin password enables +authentication, and every Server replica must share one token secret so a +token issued by any server validates on all of them. The `:?` guards fire +on every Compose subcommand, including `down`. + +```bash +( + set -eu + cd docker + command -v openssl >/dev/null 2>&1 || { echo "openssl not found" >&2; exit 1; } + [ -e .env ] || install -m 600 /dev/null .env + chmod 600 .env + # Keep appends on their own lines even if the file was hand-edited. + [ ! -s .env ] || [ -z "$(tail -c1 .env)" ] || printf '\n' >> .env + pat='^[[:space:]]*(export[[:space:]]+)?' + if ! grep -Eq "${pat}HUGEGRAPH_ADMIN_PASSWORD=" .env; then + admin_password="$(openssl rand -base64 12)" + printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\n" "${admin_password}" >> .env + unset admin_password + fi + if ! grep -Eq "${pat}HUGEGRAPH_AUTH_TOKEN_SECRET=" .env; then + token_secret="$(openssl rand -hex 32)" + printf "HUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" "${token_secret}" >> .env + unset token_secret + fi + # Parse values as data. Do not source .env as shell: a value such as + # HUGEGRAPH_ADMIN_PASSWORD=$(...) would run on the operator host. + # The optional export group is capture 1; the quoted value is capture 2. + read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env + } + if ! admin_value="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)"; then + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 + fi + if ! token_value="$(read_dotenv_value HUGEGRAPH_AUTH_TOKEN_SECRET)"; then + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must use the documented single-quoted format" >&2 + exit 1 + fi + [ -n "${admin_value}" ] || { + echo "HUGEGRAPH_ADMIN_PASSWORD must be non-empty in docker/.env" >&2 + exit 1 + } + [ "${#token_value}" -ge 32 ] || { + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must be at least 32 bytes in docker/.env" >&2 + exit 1 + } + unset admin_value token_value + # The shared cluster network. To override the name, export + # HUGEGRAPH_NETWORK in this shell before running the block — a value + # in docker/.env is read by Compose, not by this script. + net="${HUGEGRAPH_NETWORK:-hugegraph-net}" + docker network inspect "${net}" >/dev/null 2>&1 || + docker network create "${net}" + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + docker compose -f docker-compose-3pd-3store-3server.yml config --quiet +) +``` + +Then start the cluster: + ```bash cd docker -HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose-3pd-3store-3server.yml up -d +docker compose -f docker-compose-3pd-3store-3server.yml up -d -# To stop and remove all data volumes (clean restart) +# To stop and remove all data volumes (clean restart). +# The external hugegraph-net network is intentionally left in place. +# If the Hubble add-on is running, see "Hubble for the 3-Node Cluster" +# for the teardown that matches how it was started. docker compose -f docker-compose-3pd-3store-3server.yml down -v ``` +Pin a release by setting `HUGEGRAPH_VERSION` in `docker/.env` — the +cluster, the Hubble add-on, and the single-node quickstart file all read +it from there, so those versions cannot drift apart +(`docker-compose.dev.yml` builds PD/Store/Server from source and defaults +Hubble to `hugegraph/hubble:latest`; set `HUBBLE_IMAGE` to pin it). +Unpinned, the images default to `latest`; note the authenticated PD/Hubble +integration requires a release newer than `1.7.x`. On an older image the +Server ignores `PASSWORD` and `HG_SERVER_AUTH_TOKEN_SECRET`, which would +leave you an unauthenticated cluster. Two defaults keep that from passing +unnoticed: the cluster and add-on use `pull_policy: always`, so a stale +cached `latest` is refreshed on every `up -d`, and the Server healthcheck +requires an unauthenticated graph request to return `401` and an +authenticated one to return `200`. An image that does not enforce +authentication therefore never reports healthy, and `up -d --wait` fails +instead of handing you a false green. + +Each Server registers its `HG_SERVER_REST_URL` with PD. The defaults +(`server0`, `server1`, `server2`) are Docker-network names, which is correct +for Hubble and anything else attached to `hugegraph-net`, but they do not +resolve from an unrelated host network. A PD-aware client running outside +Docker will receive names it cannot reach. For that case, set +`HUGEGRAPH_SERVER0_REST_URL`, `HUGEGRAPH_SERVER1_REST_URL`, and +`HUGEGRAPH_SERVER2_REST_URL` to addresses that resolve from both the Server +containers and that client, and publish the matching host ports with +`HUGEGRAPH_SERVER_PUBLISH_HOST`. + +> [!NOTE] +> Upgrading an existing 3-node deployment: +> - Create `docker/.env` (block above) before running any Compose command +> against an older stack, `down` included. +> - The cluster now joins the pre-created `hugegraph-net` network instead of +> a per-project bridge, so the first `up -d` recreates all nine containers. +> Named data volumes are unchanged and survive the move; the orphaned +> `hugegraph-3x3_hg-net` bridge can be removed with +> `docker network rm hugegraph-3x3_hg-net`. +> - Authentication is now enabled: previously unauthenticated clients of the +> graph APIs on ports 8080–8082 will start receiving 401 responses and Review Comment: Remote clients will not see 401, they will see connection refused. This PR also moves every published port from all interfaces to `127.0.0.1`, so for the deployments this note addresses (another machine querying 8080-8082) the TCP connection fails before authentication is reached. The note is where an upgrader looks, and it does not mention `HUGEGRAPH_SERVER_PUBLISH_HOST` or `HUGEGRAPH_CONTROL_PLANE_HOST` as the way to restore reachability. Worth a line, especially given the advice just below to verify sign-in before decommissioning an existing access path. ########## docker/README.md: ########## @@ -196,11 +331,268 @@ curl http://localhost:8520/v1/health # Check Server (Graph API) curl http://localhost:8080/versions -# List registered stores via PD -curl http://localhost:8620/v1/stores +# Every PD endpoint except /v1/health, /actuator/* and /v1/prom/targets/* +# needs an Authorization header. PD only checks that the Basic-auth user is +# one of its internal service names, so the empty password below is enough +# — and it grants the full PD control plane, writes included, not just these +# reads. That is exactly why the shared network must be treated as a trust +# boundary. Without the header PD answers with an exception body, not data. +pd_auth="Authorization: Basic $(printf 'hubble:' | base64)" + +# List registered stores via PD (expect three, each "state":"Up") +curl -H "${pd_auth}" http://localhost:8620/v1/stores # List partitions -curl http://localhost:8620/v1/partitions +curl -H "${pd_auth}" http://localhost:8620/v1/partitions +``` + +Confirm authentication actually engaged — `/versions` stays open by design, +so it cannot tell you whether auth is on. A graph read without credentials +must be rejected: + +```bash +cd docker +# Expect 401 on all three replicas +for port in 8080 8081 8082; do + curl -s -o /dev/null -w "${port}: %{http_code}\n" \ + "http://localhost:${port}/graphs/hugegraph/schema/vertexlabels"; +done + +# And a signed-in read must succeed. Parse the generated single-quoted +# value without executing docker/.env as shell. Passing the credential +# through --config keeps it out of argv, where `ps` would expose it. +read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env +} +HUGEGRAPH_ADMIN_PASSWORD="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)" || { + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 +} +curl -s -o /dev/null -w '%{http_code}\n' \ + --config <(printf 'user = "admin:%s"\n' "${HUGEGRAPH_ADMIN_PASSWORD}") \ + http://localhost:8080/graphs/hugegraph/schema/vertexlabels +unset HUGEGRAPH_ADMIN_PASSWORD +``` + +`200` from the second command with `401` from all three of the first means +authentication is on and working. If the first command returns `200`, the +running image ignored `PASSWORD` and the cluster is **unauthenticated** — +the most likely cause is an image older than the release this integration +needs (see the version note in the quickstart above). + +### Running without authentication + +The cluster above authenticates, and that is the right default: it is the +configuration most people copy, and an open graph database on a reachable +port is a bad surprise. For a throwaway local trial on a trusted network you +can opt out with `docker-compose-3x3.non-auth.yml`, which drops the admin +password and the JWT secret from all three Servers. Anyone who can reach the +published ports then has full read and write access to every graph, so do not +use it anywhere else. + +```bash +cd docker + +# No docker/.env and no credential setup are needed for this mode. The two +# placeholders exist only because Compose evaluates the base file's +# required-variable guards before applying the override, which then drops +# both variables, so neither value reaches a container. Needs Compose v2.24+. +# +# Keep them inline on this command; do not write them into docker/.env. They +# are published values rather than secrets, and the token placeholder is +# deliberately shorter than the 32 bytes the Server requires, so if it ever +# reaches an authenticated Server that Server refuses to start instead of +# signing tokens with a key anyone can read here. +HUGEGRAPH_ADMIN_PASSWORD=non-auth-placeholder \ +HUGEGRAPH_AUTH_TOKEN_SECRET=non-auth-placeholder \ + docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-3x3.non-auth.yml up -d + +# Every graph request now succeeds without credentials: expect 200, not 401. +curl -s -o /dev/null -w '%{http_code}\n' \ + http://localhost:8080/graphs/hugegraph/schema/vertexlabels +``` + +Hubble needs the matching configuration, because its own default is to demand +a login. Point it at the non-auth properties file when you attach it: + +```bash +HUBBLE_PROPERTIES=./hugegraph-hubble-3x3.non-auth.properties \ + docker compose -p hugegraph-hubble -f docker-compose-hubble.yml up -d +``` + +<details> +<summary>Prompt for an AI assistant</summary> + +Copy this to an assistant instead of transcribing the commands by hand. + +```markdown +Start the Apache HugeGraph 3-node Docker cluster (3 PD, 3 Store, 3 Server) +with Hubble, from the `docker/` directory of the hugegraph repository. + +Use the authenticated default unless I say otherwise: +1. Follow "3-Node Cluster Quickstart" in docker/README.md to create + docker/.env with a generated admin password and JWT token secret, and to + create the external hugegraph-net network. +2. Start the cluster with docker-compose-3pd-3store-3server.yml, then attach + Hubble with docker-compose-hubble.yml. +3. Verify: an unauthenticated graph request returns 401, an authenticated one + returns 200, and Hubble answers on http://localhost:8088. +4. Tell me the admin password so I can sign in to Hubble. + +Do not weaken authentication to work around an error. If a Server never +becomes healthy, the image is probably older than the release this stack +needs; report that instead of disabling the healthcheck. Only if I explicitly +ask for an unauthenticated cluster, use the "Running without authentication" +section instead. +``` + +</details> + +--- + +## Hubble for the 3-Node Cluster + +`docker-compose-hubble.yml` defines only the Hubble service. It joins the +cluster's external network (`hugegraph-net` by default, override with +`HUGEGRAPH_NETWORK`) and has no `depends_on` on cluster services, so +starting, stopping, or upgrading Hubble never recreates or restarts PD, +Store, or Server containers. Hubble reads the cluster topology from +`hugegraph-hubble-3x3.properties`; adjust that file when attaching to a +cluster with different hostnames. Its `pd.server` is a single PD address +with no failover, so if that PD is down the operations view goes blind even +though the cluster still has quorum — repoint it at a surviving PD. The +operations view also reports one `SERVER` node, not three: it describes the +replica Hubble is currently talking to, not the whole Server tier. The add-on +stores Hubble's H2 database and uploaded files in named volumes +`hugegraph-hubble-db` (mounted at `/hubble/db`) and +`hugegraph-hubble-upload-files`. Those names are explicit, so the attach +and combined Compose projects share the same physical volumes. Hubble's +default H2 URL would write `/hubble/db.mv.db` (outside that mount); the +add-on sets `SPRING_DATASOURCE_URL` so the database file lives inside +`/hubble/db`. Recreating the container does not discard that state. + +Sign in at `http://localhost:8088` as `admin` with the +`HUGEGRAPH_ADMIN_PASSWORD` from `docker/.env`. Hubble binds to host +loopback by default (`HUBBLE_PUBLISH_HOST`, same caveats as the +single-node setup). + +Which flow to use: pick the attach flow when you do not have the cluster's +`docker/.env` — the add-on carries no `:?` guards, so it is the only flow +that runs without those credentials, which is what you want against a +cluster someone else started. Otherwise use the combined flow, including to +add Hubble to an already-running cluster (`up -d --no-deps hubble`). + +The two flows below create Hubble in different Compose projects, so manage +Hubble with the same flags you started it with: the attach flow always uses +`-p hugegraph-hubble -f docker-compose-hubble.yml`, the combined flow always +uses both `-f` flags. The explicit `-p` keeps the attach project independent +of the directory name and of other Compose projects. + +> [!IMPORTANT] +> Do not drop the `-p` from the attach flow. Without it Compose names the +> project after the current directory (`docker`), so Hubble starts in a +> third project that none of the commands below manage: `down` reports +> nothing to remove while `hg-hubble` keeps running, and every later `up` +> in either flow fails on the container name. If that happens, find it with +> `docker ps --filter name=hg-hubble` and remove it with +> `docker rm -f hg-hubble`. + +Run one Hubble per host: the single-node stack and both add-on flows all +publish `127.0.0.1:8088` and name their container `hg-hubble`. The two +add-on flows are therefore mutually exclusive — starting one while the +other's Hubble exists fails with a container-name conflict, so remove the +Hubble of the flow you are leaving before switching. The two directions are +not symmetric: + +```bash +cd docker +# leaving the attach flow (removes only Hubble) +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down + +# leaving the combined flow — remove ONLY Hubble; a plain `down` with both +# -f flags would stop all ten containers just to move one. +docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-hubble.yml rm -sf hubble +``` + +### Attach to a running cluster + +With the 3-node cluster already up: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml up -d +``` + +Lifecycle commands in this flow operate on Hubble alone and leave the +cluster and the external network in place: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml ps +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down +``` + +To remove everything in this flow, take down Hubble first, then the cluster: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down +docker compose -f docker-compose-3pd-3store-3server.yml down -v Review Comment: This command cannot run in the scenario the attach flow is prescribed for. The flow is recommended above "when you do not have the cluster's `docker/.env`", but this line hits the base file's `:?` guards and fails with `required variable HUGEGRAPH_AUTH_TOKEN_SECRET is missing a value`. Also, "everything" leaves both Hubble volumes behind: the attach `down` above has no `-v`, and the cluster file does not declare them. Because the names are fixed, a later deployment silently reattaches the stale H2 database. ########## .github/workflows/server-ci.yml: ########## @@ -138,6 +140,322 @@ jobs: check_compose docker/docker-compose.yml always always check_compose docker/docker-compose.dev.yml build missing + check_cluster_compose() { + local cluster="docker/docker-compose-3pd-3store-3server.yml" + local addon="docker/docker-compose-hubble.yml" + local rendered + local token_fixture=ci-test-token-secret-32-bytes-long + rendered="$(mktemp)" + if [ "${#token_fixture}" -lt 32 ]; then + echo "CI token fixture must be at least 32 bytes" >&2 + return 1 + fi + + # RETURN only: an EXIT trap would fire after this function's + # `local rendered` has gone out of scope, which `set -u` turns + # into an "unbound variable" error. A hard errexit abort can + # therefore still leak one temp file, which is acceptable on an + # ephemeral runner. + trap 'rm -f "$rendered"' RETURN + + # --env-file /dev/null on every invocation: Compose otherwise reads + # docker/.env automatically, and the quickstart tells operators to + # create one holding exactly the credentials these assertions + # control. Without it the checks pass in CI (which has no .env) but + # report false failures for anyone running them locally after + # following the quickstart, and the add-on render below — which + # deliberately unsets the variables to assert their defaults — would + # read whatever HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION that .env holds. + # Pinning an empty env file makes both renders depend only on what + # each invocation sets explicitly. + + # Both cluster credentials are required and may not be empty. + # Each case asserts the guard fired for the *intended* variable: + # a bare non-zero exit would also be produced by a YAML error, a + # renamed file, or a missing docker binary. + assert_guard() { # assert_guard <blamed-var> <description> + local var="$1" desc="$2" err + if err="$(docker compose --env-file /dev/null -f "$cluster" \ + config -q 2>&1)"; then + echo "$cluster accepted $desc" >&2 + return 1 + fi + case "$err" in + *"$var"*) : ;; + *) echo "$cluster rejected $desc, but not because of $var: $err" >&2 + return 1 ;; + esac + } + + ( unset HUGEGRAPH_ADMIN_PASSWORD + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an unset admin password" ) + ( export HUGEGRAPH_ADMIN_PASSWORD= + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an empty admin password" ) + ( unset HUGEGRAPH_AUTH_TOKEN_SECRET + export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an unset token secret" ) + ( export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + export HUGEGRAPH_AUTH_TOKEN_SECRET= + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an empty token secret" ) + # The add-on alone must define Hubble and nothing else, join the + # shared external network with its default name, and need no + # credentials or overrides. + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + -u HUGEGRAPH_NETWORK -u HUGEGRAPH_VERSION \ + -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ + -u HUBBLE_DB_VOLUME -u HUBBLE_UPLOAD_VOLUME \ + docker compose --env-file /dev/null -f "$addon" config --format json > "$rendered" + jq -e ' + (.services | keys) == ["hubble"] and + .networks."hg-net".external == true and + .networks."hg-net".name == "hugegraph-net" and + (.services.hubble.networks | has("hg-net")) and + (.services.hubble | has("depends_on") | not) and + .volumes."hg-hubble-db".name == "hugegraph-hubble-db" and + .volumes."hg-hubble-upload-files".name == + "hugegraph-hubble-upload-files" and + .services.hubble.environment.SPRING_DATASOURCE_URL == + "jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE" and + any(.services.hubble.volumes[]; + .target == "/hubble/conf/hugegraph-hubble.properties" and + (.source | endswith("hugegraph-hubble-3x3.properties"))) + and any(.services.hubble.volumes[]; + .source == "hg-hubble-db" and + .target == "/hubble/db") + and any(.services.hubble.volumes[]; + .source == "hg-hubble-upload-files" and + .target == "/hubble/upload-files") + ' "$rendered" >/dev/null + + # The cluster's own default network name must match the add-on's, + # or the attach flow and the cluster land on different networks. + # The combined render below pins an override, so it cannot catch a + # drifting default. + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + env -u HUGEGRAPH_NETWORK -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY \ + -u HUBBLE_PUBLISH_HOST \ + docker compose --env-file /dev/null -f "$cluster" \ + config --format json > "$rendered" + jq -e '.networks."hg-net".name == "hugegraph-net"' \ + "$rendered" >/dev/null + + # The combined render carries the PD-registration and auth + # settings on every server replica and keeps Hubble on loopback. + # Rendered with non-default HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION so + # CI fails if any file stops honoring the overrides (the add-on + # render above covers the defaults). + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + HUGEGRAPH_NETWORK=ci-test-net \ + HUGEGRAPH_VERSION=ci-test-tag \ + env -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ + -u HUBBLE_DB_VOLUME -u HUBBLE_UPLOAD_VOLUME \ + docker compose --env-file /dev/null -f "$cluster" -f "$addon" \ + config --format json > "$rendered" + jq -e ' + . as $root | + .name == "hugegraph-3x3" and + (.services | keys | length) == 10 and + .networks."hg-net".external == true and + .networks."hg-net".name == "ci-test-net" and + all(.services[]; .networks | has("hg-net")) and + .services.pd0.image == "hugegraph/pd:ci-test-tag" and + .services.store0.image == "hugegraph/store:ci-test-tag" and + .services.server0.image == "hugegraph/server:ci-test-tag" and + .services.hubble.image == "hugegraph/hubble:ci-test-tag" and + all(["server0", "server1", "server2"][]; + $root.services[.].environment.HG_SERVER_USE_PD == "true" and + $root.services[.].environment.HG_SERVER_CLUSTER == "hg" and + $root.services[.].environment.HG_SERVER_INIT_STORE_ENABLED == + "false" and + $root.services[.].environment.PASSWORD == + "ci-test-password" and + $root.services[.].environment.HG_SERVER_AUTH_TOKEN_SECRET == + "ci-test-token-secret-32-bytes-long") and + .services.server0.environment.HG_SERVER_REST_URL == + "http://server0:8080"; and + .services.server1.environment.HG_SERVER_REST_URL == + "http://server1:8080"; and + .services.server2.environment.HG_SERVER_REST_URL == + "http://server2:8080"; and + (.services.hubble | has("depends_on") | not) and + .services.hubble.pull_policy == "always" and + .volumes."hg-hubble-db".name == "hugegraph-hubble-db" and + .volumes."hg-hubble-upload-files".name == + "hugegraph-hubble-upload-files" and + .services.hubble.environment.SPRING_DATASOURCE_URL == + "jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE" and + (.services.hubble.healthcheck.test[1] | + contains("http://127.0.0.1:8088/about";) and + contains("\"status\":200") and + contains("\"name\":\"hugegraph-hubble\"")) and + any(.services.hubble.ports[]; + .target == 8088 and .published == "8088" and + .host_ip == "127.0.0.1") and + all(["pd0", "pd1", "pd2", "store0", "store1", "store2", + "server0", "server1", "server2"][]; + all($root.services[.].ports[]; .host_ip == "127.0.0.1")) + ' "$rendered" >/dev/null + + # Hubble's properties file is mounted, not rendered, so Compose + # validation alone cannot catch it drifting from the services it + # describes. Tie the two together: renaming a service, changing a + # container hostname, or moving a REST port must be reflected in + # both places or CI fails here. Values are derived from the + # rendered model (hostnames and ports included) so the assertions + # cannot silently agree with a stale file. + local props="docker/hugegraph-hubble-3x3.properties" + local pd_peers store_targets cluster_name pd_rest + assert_props() { # assert_props <exact-line> <what-it-must-match> + grep -Fqx "$1" "$props" || + { echo "$props: expected line '$1' ($2)" >&2; return 1; } + } + cluster_name="$(jq -r '.services.server0.environment.HG_SERVER_CLUSTER' \ + "$rendered")" + pd_peers="$(jq -r '.services.server0.environment.HG_SERVER_PD_PEERS' \ + "$rendered")" + store_targets="[$(jq -r '[.services | to_entries[] + | select(.key | startswith("store")) + | "http://"; + .value.hostname + ":" + + (.value.environment.HG_STORE_REST_PORT)] + | sort | join(",")' "$rendered")]" + pd_rest="$(jq -r --arg h "$(printf '%s' "${pd_peers}" | cut -d, -f1 | cut -d: -f1)" \ + '.services | to_entries[] + | select(.value.hostname == $h) + | .value.hostname + ":" + .value.environment.HG_PD_REST_PORT' \ + "$rendered")" + assert_props "cluster=${cluster_name}" "matches HG_SERVER_CLUSTER" + assert_props "pd.enabled=true" "keeps Hubble in PD mode" + assert_props "pd.peers=${pd_peers}" "matches HG_SERVER_PD_PEERS" + assert_props "pd.server=${pd_rest}" "names a real PD REST endpoint" + assert_props "operations.store.allowed_targets=${store_targets}" \ + "lists every Store REST endpoint" + + # The documented opt-out must actually opt out: no admin password, + # no shared token secret, and a healthcheck that does not demand + # the 401/200 pair the authenticated stack requires. A silent + # failure here would leave users on an unauthenticated cluster + # that still advertises itself as authenticated. + # Use the exact placeholders the README documents, including the + # deliberately short token: it must never reach a container, and + # the assertions below are what prove the override drops it. + local non_auth="docker/docker-compose-3x3.non-auth.yml" + HUGEGRAPH_ADMIN_PASSWORD=non-auth-placeholder \ + HUGEGRAPH_AUTH_TOKEN_SECRET=non-auth-placeholder \ + docker compose --env-file /dev/null -f "$cluster" -f "$non_auth" \ + config --format json > "$rendered" + jq -e ' + . as $root | + all(["server0", "server1", "server2"][]; + ($root.services[.].environment | has("PASSWORD") | not) and + ($root.services[.].environment + | has("HG_SERVER_AUTH_TOKEN_SECRET") | not) and + ($root.services[.].healthcheck.test[1] + | contains("/versions") and (contains("401") | not))) + ' "$rendered" >/dev/null + grep -Fqx "auth.enabled=false" \ + docker/hugegraph-hubble-3x3.non-auth.properties + } + + check_cluster_compose + + - name: Run distributed Compose auth and attach smoke test + if: ${{ env.BACKEND == 'rocksdb' }} + run: | + set -euo pipefail + command -v docker >/dev/null 2>&1 + docker compose version >/dev/null + docker info >/dev/null + + cluster="docker/docker-compose-3pd-3store-3server.yml" + addon="docker/docker-compose-hubble.yml" + run_id="${GITHUB_RUN_ID:-local}" + network="hugegraph-ci-${run_id}" + db_volume="hugegraph-ci-${run_id}-db" + upload_volume="hugegraph-ci-${run_id}-uploads" + admin_password='ci-smoke-admin-password' + token_secret='ci-smoke-token-secret-32-bytes-long' + + # Every Compose call needs the same environment, and cleanup must run + # even when an assertion below fails. + export HUGEGRAPH_NETWORK="$network" + export HUGEGRAPH_ADMIN_PASSWORD="$admin_password" + export HUGEGRAPH_AUTH_TOKEN_SECRET="$token_secret" + export HUBBLE_DB_VOLUME="$db_volume" + export HUBBLE_UPLOAD_VOLUME="$upload_volume" + + cleanup() { + docker compose --env-file /dev/null -f "$cluster" -f "$addon" \ + down -v --remove-orphans >/dev/null 2>&1 || true + docker compose --env-file /dev/null -p "hugegraph-ci-attach-${run_id}" \ + -f "$addon" down -v >/dev/null 2>&1 || true + docker network rm "$network" >/dev/null 2>&1 || true + docker volume rm "$db_volume" "$upload_volume" >/dev/null 2>&1 || true + } + trap cleanup EXIT + docker network create "$network" + + # The cluster must come up healthy on its own. Its Server healthcheck + # already proves 401/200, so --wait failing here means the images do + # not enforce the authentication this stack configures. + docker compose --env-file /dev/null -f "$cluster" up -d --wait \ Review Comment: This is the part of the CI addition I would most want changed. The smoke test boots `hugegraph/{pd,store,server}:latest` from Docker Hub with `pull_policy: always` and then asserts the 401/200 behavior. Nothing in the job builds the images from the PR under test, so the step is really asserting a property of the published `latest` tag: it goes red on unrelated PRs whenever `latest` lags master, the registry is slow, or the runner gets rate-limited, and it would stay green on a PR that broke the entrypoint. Worth noting the runner budget too: this is `ubuntu-22.04`, and the step boots ten JVM containers with `--wait-timeout 420` twice plus 180, in the same job that already ran a rocksdb backend and a Maven compile, against a README that asks for 12 GB for the nine cluster JVMs alone. A separate job needing only checkout and Docker would parallelize it and give the containers the full runner. Two related coverage gaps while you are here: the Hubble-to-cluster path is never exercised (PD and Store are probed from the runner, and Hubble only via its own `/about`, which passes even if Hubble can reach nothing), and the non-auth flow is only ever rendered, never booted. ########## .github/workflows/server-ci.yml: ########## @@ -138,6 +140,322 @@ jobs: check_compose docker/docker-compose.yml always always check_compose docker/docker-compose.dev.yml build missing + check_cluster_compose() { + local cluster="docker/docker-compose-3pd-3store-3server.yml" + local addon="docker/docker-compose-hubble.yml" + local rendered + local token_fixture=ci-test-token-secret-32-bytes-long + rendered="$(mktemp)" + if [ "${#token_fixture}" -lt 32 ]; then + echo "CI token fixture must be at least 32 bytes" >&2 + return 1 + fi + + # RETURN only: an EXIT trap would fire after this function's + # `local rendered` has gone out of scope, which `set -u` turns + # into an "unbound variable" error. A hard errexit abort can + # therefore still leak one temp file, which is acceptable on an + # ephemeral runner. + trap 'rm -f "$rendered"' RETURN + + # --env-file /dev/null on every invocation: Compose otherwise reads + # docker/.env automatically, and the quickstart tells operators to + # create one holding exactly the credentials these assertions + # control. Without it the checks pass in CI (which has no .env) but + # report false failures for anyone running them locally after + # following the quickstart, and the add-on render below — which + # deliberately unsets the variables to assert their defaults — would + # read whatever HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION that .env holds. + # Pinning an empty env file makes both renders depend only on what + # each invocation sets explicitly. + + # Both cluster credentials are required and may not be empty. + # Each case asserts the guard fired for the *intended* variable: + # a bare non-zero exit would also be produced by a YAML error, a + # renamed file, or a missing docker binary. + assert_guard() { # assert_guard <blamed-var> <description> + local var="$1" desc="$2" err + if err="$(docker compose --env-file /dev/null -f "$cluster" \ + config -q 2>&1)"; then + echo "$cluster accepted $desc" >&2 + return 1 + fi + case "$err" in + *"$var"*) : ;; + *) echo "$cluster rejected $desc, but not because of $var: $err" >&2 + return 1 ;; + esac + } + + ( unset HUGEGRAPH_ADMIN_PASSWORD + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an unset admin password" ) + ( export HUGEGRAPH_ADMIN_PASSWORD= + export HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" + assert_guard HUGEGRAPH_ADMIN_PASSWORD "an empty admin password" ) + ( unset HUGEGRAPH_AUTH_TOKEN_SECRET + export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an unset token secret" ) + ( export HUGEGRAPH_ADMIN_PASSWORD=ci-test-password + export HUGEGRAPH_AUTH_TOKEN_SECRET= + assert_guard HUGEGRAPH_AUTH_TOKEN_SECRET "an empty token secret" ) + # The add-on alone must define Hubble and nothing else, join the + # shared external network with its default name, and need no + # credentials or overrides. + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + -u HUGEGRAPH_NETWORK -u HUGEGRAPH_VERSION \ + -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ + -u HUBBLE_DB_VOLUME -u HUBBLE_UPLOAD_VOLUME \ + docker compose --env-file /dev/null -f "$addon" config --format json > "$rendered" + jq -e ' + (.services | keys) == ["hubble"] and + .networks."hg-net".external == true and + .networks."hg-net".name == "hugegraph-net" and + (.services.hubble.networks | has("hg-net")) and + (.services.hubble | has("depends_on") | not) and + .volumes."hg-hubble-db".name == "hugegraph-hubble-db" and + .volumes."hg-hubble-upload-files".name == + "hugegraph-hubble-upload-files" and + .services.hubble.environment.SPRING_DATASOURCE_URL == + "jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE" and + any(.services.hubble.volumes[]; + .target == "/hubble/conf/hugegraph-hubble.properties" and + (.source | endswith("hugegraph-hubble-3x3.properties"))) + and any(.services.hubble.volumes[]; + .source == "hg-hubble-db" and + .target == "/hubble/db") + and any(.services.hubble.volumes[]; + .source == "hg-hubble-upload-files" and + .target == "/hubble/upload-files") + ' "$rendered" >/dev/null + + # The cluster's own default network name must match the add-on's, + # or the attach flow and the cluster land on different networks. + # The combined render below pins an override, so it cannot catch a + # drifting default. + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + env -u HUGEGRAPH_NETWORK -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY \ + -u HUBBLE_PUBLISH_HOST \ + docker compose --env-file /dev/null -f "$cluster" \ + config --format json > "$rendered" + jq -e '.networks."hg-net".name == "hugegraph-net"' \ + "$rendered" >/dev/null + + # The combined render carries the PD-registration and auth + # settings on every server replica and keeps Hubble on loopback. + # Rendered with non-default HUGEGRAPH_NETWORK/HUGEGRAPH_VERSION so + # CI fails if any file stops honoring the overrides (the add-on + # render above covers the defaults). + HUGEGRAPH_ADMIN_PASSWORD=ci-test-password \ + HUGEGRAPH_AUTH_TOKEN_SECRET="${token_fixture}" \ + HUGEGRAPH_NETWORK=ci-test-net \ + HUGEGRAPH_VERSION=ci-test-tag \ + env -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ + -u HUBBLE_DB_VOLUME -u HUBBLE_UPLOAD_VOLUME \ + docker compose --env-file /dev/null -f "$cluster" -f "$addon" \ + config --format json > "$rendered" + jq -e ' + . as $root | + .name == "hugegraph-3x3" and + (.services | keys | length) == 10 and + .networks."hg-net".external == true and + .networks."hg-net".name == "ci-test-net" and + all(.services[]; .networks | has("hg-net")) and + .services.pd0.image == "hugegraph/pd:ci-test-tag" and + .services.store0.image == "hugegraph/store:ci-test-tag" and + .services.server0.image == "hugegraph/server:ci-test-tag" and + .services.hubble.image == "hugegraph/hubble:ci-test-tag" and + all(["server0", "server1", "server2"][]; + $root.services[.].environment.HG_SERVER_USE_PD == "true" and + $root.services[.].environment.HG_SERVER_CLUSTER == "hg" and + $root.services[.].environment.HG_SERVER_INIT_STORE_ENABLED == + "false" and + $root.services[.].environment.PASSWORD == + "ci-test-password" and + $root.services[.].environment.HG_SERVER_AUTH_TOKEN_SECRET == + "ci-test-token-secret-32-bytes-long") and + .services.server0.environment.HG_SERVER_REST_URL == + "http://server0:8080"; and + .services.server1.environment.HG_SERVER_REST_URL == + "http://server1:8080"; and + .services.server2.environment.HG_SERVER_REST_URL == + "http://server2:8080"; and + (.services.hubble | has("depends_on") | not) and + .services.hubble.pull_policy == "always" and + .volumes."hg-hubble-db".name == "hugegraph-hubble-db" and + .volumes."hg-hubble-upload-files".name == + "hugegraph-hubble-upload-files" and + .services.hubble.environment.SPRING_DATASOURCE_URL == + "jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE" and + (.services.hubble.healthcheck.test[1] | + contains("http://127.0.0.1:8088/about";) and + contains("\"status\":200") and + contains("\"name\":\"hugegraph-hubble\"")) and + any(.services.hubble.ports[]; + .target == 8088 and .published == "8088" and + .host_ip == "127.0.0.1") and + all(["pd0", "pd1", "pd2", "store0", "store1", "store2", + "server0", "server1", "server2"][]; + all($root.services[.].ports[]; .host_ip == "127.0.0.1")) + ' "$rendered" >/dev/null + + # Hubble's properties file is mounted, not rendered, so Compose + # validation alone cannot catch it drifting from the services it + # describes. Tie the two together: renaming a service, changing a + # container hostname, or moving a REST port must be reflected in + # both places or CI fails here. Values are derived from the + # rendered model (hostnames and ports included) so the assertions + # cannot silently agree with a stale file. + local props="docker/hugegraph-hubble-3x3.properties" + local pd_peers store_targets cluster_name pd_rest + assert_props() { # assert_props <exact-line> <what-it-must-match> + grep -Fqx "$1" "$props" || + { echo "$props: expected line '$1' ($2)" >&2; return 1; } + } + cluster_name="$(jq -r '.services.server0.environment.HG_SERVER_CLUSTER' \ + "$rendered")" + pd_peers="$(jq -r '.services.server0.environment.HG_SERVER_PD_PEERS' \ + "$rendered")" + store_targets="[$(jq -r '[.services | to_entries[] + | select(.key | startswith("store")) + | "http://"; + .value.hostname + ":" + + (.value.environment.HG_STORE_REST_PORT)] + | sort | join(",")' "$rendered")]" + pd_rest="$(jq -r --arg h "$(printf '%s' "${pd_peers}" | cut -d, -f1 | cut -d: -f1)" \ + '.services | to_entries[] + | select(.value.hostname == $h) + | .value.hostname + ":" + .value.environment.HG_PD_REST_PORT' \ + "$rendered")" + assert_props "cluster=${cluster_name}" "matches HG_SERVER_CLUSTER" + assert_props "pd.enabled=true" "keeps Hubble in PD mode" + assert_props "pd.peers=${pd_peers}" "matches HG_SERVER_PD_PEERS" + assert_props "pd.server=${pd_rest}" "names a real PD REST endpoint" + assert_props "operations.store.allowed_targets=${store_targets}" \ + "lists every Store REST endpoint" + + # The documented opt-out must actually opt out: no admin password, + # no shared token secret, and a healthcheck that does not demand + # the 401/200 pair the authenticated stack requires. A silent + # failure here would leave users on an unauthenticated cluster + # that still advertises itself as authenticated. + # Use the exact placeholders the README documents, including the + # deliberately short token: it must never reach a container, and + # the assertions below are what prove the override drops it. + local non_auth="docker/docker-compose-3x3.non-auth.yml" + HUGEGRAPH_ADMIN_PASSWORD=non-auth-placeholder \ + HUGEGRAPH_AUTH_TOKEN_SECRET=non-auth-placeholder \ + docker compose --env-file /dev/null -f "$cluster" -f "$non_auth" \ + config --format json > "$rendered" + jq -e ' + . as $root | + all(["server0", "server1", "server2"][]; + ($root.services[.].environment | has("PASSWORD") | not) and + ($root.services[.].environment + | has("HG_SERVER_AUTH_TOKEN_SECRET") | not) and + ($root.services[.].healthcheck.test[1] + | contains("/versions") and (contains("401") | not))) + ' "$rendered" >/dev/null + grep -Fqx "auth.enabled=false" \ Review Comment: The non-auth properties file gets only this one grep, while `assert_props` ties the authenticated file's topology to the rendered model. Its `pd.peers`, `pd.server`, `operations.store.allowed_targets` and `cluster` can therefore drift from the compose files with CI green, and since the non-auth flow is never booted, users are the detection mechanism. All five expected lines are already present verbatim in that file, so parameterising `assert_props` on the path and running it twice would cover it. Related: `hugegraph-hubble-3x3.properties` never pins `auth.enabled=true`. The file's own header warns that absent keys fall back to code defaults a future release can change, and the non-auth twin pins its value with this grep behind it. Pinning the positive case too, with a matching grep, makes that drift impossible. ########## docker/README.md: ########## @@ -196,11 +331,268 @@ curl http://localhost:8520/v1/health # Check Server (Graph API) curl http://localhost:8080/versions -# List registered stores via PD -curl http://localhost:8620/v1/stores +# Every PD endpoint except /v1/health, /actuator/* and /v1/prom/targets/* +# needs an Authorization header. PD only checks that the Basic-auth user is +# one of its internal service names, so the empty password below is enough +# — and it grants the full PD control plane, writes included, not just these +# reads. That is exactly why the shared network must be treated as a trust +# boundary. Without the header PD answers with an exception body, not data. +pd_auth="Authorization: Basic $(printf 'hubble:' | base64)" + +# List registered stores via PD (expect three, each "state":"Up") +curl -H "${pd_auth}" http://localhost:8620/v1/stores # List partitions -curl http://localhost:8620/v1/partitions +curl -H "${pd_auth}" http://localhost:8620/v1/partitions +``` + +Confirm authentication actually engaged — `/versions` stays open by design, +so it cannot tell you whether auth is on. A graph read without credentials +must be rejected: + +```bash +cd docker +# Expect 401 on all three replicas +for port in 8080 8081 8082; do + curl -s -o /dev/null -w "${port}: %{http_code}\n" \ + "http://localhost:${port}/graphs/hugegraph/schema/vertexlabels"; +done + +# And a signed-in read must succeed. Parse the generated single-quoted +# value without executing docker/.env as shell. Passing the credential +# through --config keeps it out of argv, where `ps` would expose it. +read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env +} +HUGEGRAPH_ADMIN_PASSWORD="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)" || { + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 +} +curl -s -o /dev/null -w '%{http_code}\n' \ + --config <(printf 'user = "admin:%s"\n' "${HUGEGRAPH_ADMIN_PASSWORD}") \ + http://localhost:8080/graphs/hugegraph/schema/vertexlabels +unset HUGEGRAPH_ADMIN_PASSWORD +``` + +`200` from the second command with `401` from all three of the first means +authentication is on and working. If the first command returns `200`, the +running image ignored `PASSWORD` and the cluster is **unauthenticated** — +the most likely cause is an image older than the release this integration +needs (see the version note in the quickstart above). + +### Running without authentication + +The cluster above authenticates, and that is the right default: it is the +configuration most people copy, and an open graph database on a reachable +port is a bad surprise. For a throwaway local trial on a trusted network you +can opt out with `docker-compose-3x3.non-auth.yml`, which drops the admin +password and the JWT secret from all three Servers. Anyone who can reach the +published ports then has full read and write access to every graph, so do not +use it anywhere else. + +```bash +cd docker + +# No docker/.env and no credential setup are needed for this mode. The two +# placeholders exist only because Compose evaluates the base file's +# required-variable guards before applying the override, which then drops +# both variables, so neither value reaches a container. Needs Compose v2.24+. +# +# Keep them inline on this command; do not write them into docker/.env. They +# are published values rather than secrets, and the token placeholder is +# deliberately shorter than the 32 bytes the Server requires, so if it ever +# reaches an authenticated Server that Server refuses to start instead of +# signing tokens with a key anyone can read here. +HUGEGRAPH_ADMIN_PASSWORD=non-auth-placeholder \ +HUGEGRAPH_AUTH_TOKEN_SECRET=non-auth-placeholder \ + docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-3x3.non-auth.yml up -d + +# Every graph request now succeeds without credentials: expect 200, not 401. +curl -s -o /dev/null -w '%{http_code}\n' \ + http://localhost:8080/graphs/hugegraph/schema/vertexlabels +``` + +Hubble needs the matching configuration, because its own default is to demand +a login. Point it at the non-auth properties file when you attach it: + +```bash +HUBBLE_PROPERTIES=./hugegraph-hubble-3x3.non-auth.properties \ + docker compose -p hugegraph-hubble -f docker-compose-hubble.yml up -d +``` + +<details> +<summary>Prompt for an AI assistant</summary> + +Copy this to an assistant instead of transcribing the commands by hand. + +```markdown +Start the Apache HugeGraph 3-node Docker cluster (3 PD, 3 Store, 3 Server) +with Hubble, from the `docker/` directory of the hugegraph repository. + +Use the authenticated default unless I say otherwise: +1. Follow "3-Node Cluster Quickstart" in docker/README.md to create + docker/.env with a generated admin password and JWT token secret, and to + create the external hugegraph-net network. +2. Start the cluster with docker-compose-3pd-3store-3server.yml, then attach + Hubble with docker-compose-hubble.yml. +3. Verify: an unauthenticated graph request returns 401, an authenticated one + returns 200, and Hubble answers on http://localhost:8088. +4. Tell me the admin password so I can sign in to Hubble. + +Do not weaken authentication to work around an error. If a Server never +becomes healthy, the image is probably older than the release this stack +needs; report that instead of disabling the healthcheck. Only if I explicitly +ask for an unauthenticated cluster, use the "Running without authentication" +section instead. +``` + +</details> + +--- + +## Hubble for the 3-Node Cluster + +`docker-compose-hubble.yml` defines only the Hubble service. It joins the +cluster's external network (`hugegraph-net` by default, override with +`HUGEGRAPH_NETWORK`) and has no `depends_on` on cluster services, so +starting, stopping, or upgrading Hubble never recreates or restarts PD, +Store, or Server containers. Hubble reads the cluster topology from +`hugegraph-hubble-3x3.properties`; adjust that file when attaching to a +cluster with different hostnames. Its `pd.server` is a single PD address +with no failover, so if that PD is down the operations view goes blind even +though the cluster still has quorum — repoint it at a surviving PD. The +operations view also reports one `SERVER` node, not three: it describes the +replica Hubble is currently talking to, not the whole Server tier. The add-on +stores Hubble's H2 database and uploaded files in named volumes +`hugegraph-hubble-db` (mounted at `/hubble/db`) and +`hugegraph-hubble-upload-files`. Those names are explicit, so the attach +and combined Compose projects share the same physical volumes. Hubble's +default H2 URL would write `/hubble/db.mv.db` (outside that mount); the +add-on sets `SPRING_DATASOURCE_URL` so the database file lives inside +`/hubble/db`. Recreating the container does not discard that state. + +Sign in at `http://localhost:8088` as `admin` with the +`HUGEGRAPH_ADMIN_PASSWORD` from `docker/.env`. Hubble binds to host +loopback by default (`HUBBLE_PUBLISH_HOST`, same caveats as the +single-node setup). + +Which flow to use: pick the attach flow when you do not have the cluster's +`docker/.env` — the add-on carries no `:?` guards, so it is the only flow +that runs without those credentials, which is what you want against a +cluster someone else started. Otherwise use the combined flow, including to +add Hubble to an already-running cluster (`up -d --no-deps hubble`). + +The two flows below create Hubble in different Compose projects, so manage +Hubble with the same flags you started it with: the attach flow always uses +`-p hugegraph-hubble -f docker-compose-hubble.yml`, the combined flow always +uses both `-f` flags. The explicit `-p` keeps the attach project independent +of the directory name and of other Compose projects. + +> [!IMPORTANT] +> Do not drop the `-p` from the attach flow. Without it Compose names the +> project after the current directory (`docker`), so Hubble starts in a +> third project that none of the commands below manage: `down` reports +> nothing to remove while `hg-hubble` keeps running, and every later `up` +> in either flow fails on the container name. If that happens, find it with +> `docker ps --filter name=hg-hubble` and remove it with +> `docker rm -f hg-hubble`. + +Run one Hubble per host: the single-node stack and both add-on flows all +publish `127.0.0.1:8088` and name their container `hg-hubble`. The two +add-on flows are therefore mutually exclusive — starting one while the +other's Hubble exists fails with a container-name conflict, so remove the +Hubble of the flow you are leaving before switching. The two directions are +not symmetric: + +```bash +cd docker +# leaving the attach flow (removes only Hubble) +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down + +# leaving the combined flow — remove ONLY Hubble; a plain `down` with both +# -f flags would stop all ten containers just to move one. +docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-hubble.yml rm -sf hubble +``` + +### Attach to a running cluster + +With the 3-node cluster already up: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml up -d +``` + +Lifecycle commands in this flow operate on Hubble alone and leave the +cluster and the external network in place: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml ps +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down +``` + +To remove everything in this flow, take down Hubble first, then the cluster: + +```bash +cd docker +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml down +docker compose -f docker-compose-3pd-3store-3server.yml down -v +``` + +### Fresh cluster plus Hubble in one command + +After the one-time network and `docker/.env` setup from the quickstart: + +```bash +cd docker +docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-hubble.yml up -d +``` + +Hubble has no startup dependency on the cluster, so it reports healthy while +PD, Store, and Server are still forming the cluster; wait until every service +shows healthy before signing in: + +```bash +cd docker +docker compose -f docker-compose-3pd-3store-3server.yml \ + -f docker-compose-hubble.yml ps +``` + +In this flow Hubble belongs to the cluster project — use the same pair of +`-f` flags for `ps`, `stop`, and `down`. The attach-flow `ps`/`stop`/`down` +commands manage a different, empty project and do nothing here, and the +cluster-only quickstart commands treat this Hubble as an orphan container +(`--remove-orphans` would delete it) — always pass both `-f` flags. + +### Local Hubble image for development + +Build the Hubble image from `hugegraph-toolchain` source, then replace only +the Hubble container. In the attach flow: + +```bash +cd docker +HUBBLE_IMAGE=local/hugegraph-hubble:dev \ +HUBBLE_PULL_POLICY=never \ +docker compose -p hugegraph-hubble -f docker-compose-hubble.yml up -d Review Comment: `HUBBLE_PROPERTIES` is not sticky, and this command drops it. Someone running the non-auth stack attached Hubble with `HUBBLE_PROPERTIES=./hugegraph-hubble-3x3.non-auth.properties`; running this dev-image command afterwards changes the bind source back to the default auth properties, so Compose recreates the container and Hubble comes back demanding a login against Servers that have no users configured. The same applies to any later plain `up -d` in that flow. Worth carrying `HUBBLE_PROPERTIES` through the commands in this section, or noting that it has to be repeated every time. ########## docker/README.md: ########## @@ -168,14 +175,142 @@ To validate local images without Compose replacing them with remote `latest`: ## 3-Node Cluster Quickstart +The cluster and the Hubble add-on share one named Docker network so Hubble +can attach to a running cluster without touching it. Treat that network as a +trust boundary: only the Server layer performs real authentication. Store +serves its control APIs with no credentials at all, and while PD's REST +control APIs do require an `Authorization` header, PD only checks that the +Basic-auth *user* is one of its internal service names and never validates +the password — so any client on the network can read and drive them. Note +that this stack depends on that behaviour: Hubble reaches PD as the service +name `hubble` with an empty password, so tightening PD's credential check +would also break Hubble's operations view. Because of that, PD and Store +control-plane ports and the three Server REST ports bind to `127.0.0.1` by +default, so an unauthenticated control plane is not reachable from outside the +host. Publish them more widely only by setting `HUGEGRAPH_CONTROL_PLANE_HOST` +or `HUGEGRAPH_SERVER_PUBLISH_HOST`, and only behind a network ACL or TLS +terminator you control. Any container on the host can still join the network +by declaring the well-known name. One-time setup: +write the required credentials to a mode-600 `docker/.env` and create the +network. +The cluster file requires both credentials — the admin password enables +authentication, and every Server replica must share one token secret so a +token issued by any server validates on all of them. The `:?` guards fire +on every Compose subcommand, including `down`. + +```bash +( + set -eu + cd docker + command -v openssl >/dev/null 2>&1 || { echo "openssl not found" >&2; exit 1; } + [ -e .env ] || install -m 600 /dev/null .env + chmod 600 .env + # Keep appends on their own lines even if the file was hand-edited. + [ ! -s .env ] || [ -z "$(tail -c1 .env)" ] || printf '\n' >> .env + pat='^[[:space:]]*(export[[:space:]]+)?' + if ! grep -Eq "${pat}HUGEGRAPH_ADMIN_PASSWORD=" .env; then + admin_password="$(openssl rand -base64 12)" + printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\n" "${admin_password}" >> .env + unset admin_password + fi + if ! grep -Eq "${pat}HUGEGRAPH_AUTH_TOKEN_SECRET=" .env; then + token_secret="$(openssl rand -hex 32)" + printf "HUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" "${token_secret}" >> .env + unset token_secret + fi + # Parse values as data. Do not source .env as shell: a value such as + # HUGEGRAPH_ADMIN_PASSWORD=$(...) would run on the operator host. + # The optional export group is capture 1; the quoted value is capture 2. + read_dotenv_value() { + key="$1" + key_pattern="^[[:space:]]*(export[[:space:]]+)?${key}=" + count="$(grep -Ec "${key_pattern}" .env || true)" + [ "${count}" -eq 1 ] || return 1 + sed -nE "s/${key_pattern}'([^']*)'[[:space:]]*$/\\2/p" .env + } + if ! admin_value="$(read_dotenv_value HUGEGRAPH_ADMIN_PASSWORD)"; then + echo "HUGEGRAPH_ADMIN_PASSWORD must use the documented single-quoted format" >&2 + exit 1 + fi + if ! token_value="$(read_dotenv_value HUGEGRAPH_AUTH_TOKEN_SECRET)"; then + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must use the documented single-quoted format" >&2 + exit 1 + fi + [ -n "${admin_value}" ] || { + echo "HUGEGRAPH_ADMIN_PASSWORD must be non-empty in docker/.env" >&2 + exit 1 + } + [ "${#token_value}" -ge 32 ] || { + echo "HUGEGRAPH_AUTH_TOKEN_SECRET must be at least 32 bytes in docker/.env" >&2 + exit 1 + } + unset admin_value token_value + # The shared cluster network. To override the name, export + # HUGEGRAPH_NETWORK in this shell before running the block — a value + # in docker/.env is read by Compose, not by this script. + net="${HUGEGRAPH_NETWORK:-hugegraph-net}" + docker network inspect "${net}" >/dev/null 2>&1 || + docker network create "${net}" + env -u HUGEGRAPH_ADMIN_PASSWORD -u HUGEGRAPH_AUTH_TOKEN_SECRET \ + docker compose -f docker-compose-3pd-3store-3server.yml config --quiet +) +``` + +Then start the cluster: + ```bash cd docker -HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose-3pd-3store-3server.yml up -d +docker compose -f docker-compose-3pd-3store-3server.yml up -d -# To stop and remove all data volumes (clean restart) +# To stop and remove all data volumes (clean restart). +# The external hugegraph-net network is intentionally left in place. +# If the Hubble add-on is running, see "Hubble for the 3-Node Cluster" +# for the teardown that matches how it was started. docker compose -f docker-compose-3pd-3store-3server.yml down -v ``` +Pin a release by setting `HUGEGRAPH_VERSION` in `docker/.env` — the +cluster, the Hubble add-on, and the single-node quickstart file all read +it from there, so those versions cannot drift apart +(`docker-compose.dev.yml` builds PD/Store/Server from source and defaults +Hubble to `hugegraph/hubble:latest`; set `HUBBLE_IMAGE` to pin it). +Unpinned, the images default to `latest`; note the authenticated PD/Hubble +integration requires a release newer than `1.7.x`. On an older image the +Server ignores `PASSWORD` and `HG_SERVER_AUTH_TOKEN_SECRET`, which would +leave you an unauthenticated cluster. Two defaults keep that from passing +unnoticed: the cluster and add-on use `pull_policy: always`, so a stale +cached `latest` is refreshed on every `up -d`, and the Server healthcheck +requires an unauthenticated graph request to return `401` and an +authenticated one to return `200`. An image that does not enforce +authentication therefore never reports healthy, and `up -d --wait` fails Review Comment: The safety net promised here does not cover every file this paragraph claims it does. It opens by saying the cluster, the add-on and the single-node quickstart all read `HUGEGRAPH_VERSION` from `docker/.env`, then concludes that an image which does not enforce authentication "never reports healthy". Only the cluster file has the 401/200 check. The add-on ships no Server, so the sentence does not apply to it; the single-node file is the gap. Its Server healthcheck is still `curl -fsS http://server:8080/versions`, and `/versions` is open by design, as this README says a few lines down. A reader who pins `HUGEGRAPH_VERSION` exactly as this paragraph instructs gets, for the single-node file, the false green it promises to prevent. Either scope the sentence to the cluster file or carry the same check into the single-node file. Worth naming 1.8.0 here rather than "newer than `1.7.x`". ########## docker/docker-compose-hubble.yml: ########## @@ -0,0 +1,64 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Hubble add-on for the distributed cluster defined in +# docker-compose-3pd-3store-3server.yml. Requires the pre-created cluster +# network (docker network create hugegraph-net). See "Hubble for the +# 3-Node Cluster" in docker/README.md for the attach and combined flows. + +networks: + hg-net: + external: true + name: ${HUGEGRAPH_NETWORK:-hugegraph-net} + +volumes: + hg-hubble-db: + # Explicit names so attach (-p hugegraph-hubble) and combined + # (project hugegraph-3x3) share the same physical volumes. + name: ${HUBBLE_DB_VOLUME:-hugegraph-hubble-db} + hg-hubble-upload-files: + name: ${HUBBLE_UPLOAD_VOLUME:-hugegraph-hubble-upload-files} + +services: + hubble: + # Pin a release via HUGEGRAPH_VERSION in docker/.env; unset, the image + # tag defaults to latest. Pull by default so an older cached image cannot + # silently satisfy an auth-sensitive deployment. + image: ${HUBBLE_IMAGE:-hugegraph/hubble:${HUGEGRAPH_VERSION:-latest}} + pull_policy: ${HUBBLE_PULL_POLICY:-always} + container_name: hg-hubble + hostname: hubble + restart: unless-stopped + networks: [hg-net] + ports: + - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088" + environment: + # Image default jdbc:h2:file:./db writes /hubble/db.mv.db, outside + # the /hubble/db volume. Point H2 at a file inside the mount. + SPRING_DATASOURCE_URL: jdbc:h2:file:./db/hubble;DB_CLOSE_ON_EXIT=FALSE Review Comment: Nit: this fixes the H2 path with the same working-directory-relative mechanism that caused the original problem. It lands inside the volume only while the container's working directory stays `/hubble` (which it is today), so an image that changes WORKDIR silently writes the database outside the mount again. ```suggestion SPRING_DATASOURCE_URL: jdbc:h2:file:/hubble/db/hubble;DB_CLOSE_ON_EXIT=FALSE ``` Anchored to the mount target declared just below under `volumes:`. The two CI assertions need the same string. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
