This is an automated email from the ASF dual-hosted git repository.
imbajin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hugegraph.git
The following commit(s) were added to refs/heads/master by this push:
new 60c8803d5 fix(pd): validate REST credentials and return 401 on refusal
(#3189)
60c8803d5 is described below
commit 60c8803d5a85943eaa05e74eb37fe7d57b2256a0
Author: KAI <[email protected]>
AuthorDate: Wed Sep 9 19:38:59 2026 +0530
fix(pd): validate REST credentials and return 401 on refusal (#3189)
PD's REST interceptor decodes the Basic credential, keeps only the part
before the colon, and checks it against the fixed set hg, store, hubble,
vermeer. Any of those names with any password, including an empty one, is
treated as an internal component; the password is never read. Separately,
RestAuthentication.preHandle writes an error body without calling setStatus, so
success, refusal and missing credential all return HTTP 200 and nothing keyed
on a status code (monitors, curl -f, t [...]
---------
Co-authored-by: imbajin <[email protected]>
---
.github/workflows/pd-store-ci.yml | 8 +
.gitignore | 2 +
docker/README.md | 103 ++++------
...-ha.properties => hstore-ha.properties.example} | 6 +
...hstore.properties => hstore.properties.example} | 6 +
docker/docker-compose-3pd-3store-3server.yml | 15 +-
docker/docker-compose-hstore.yml | 13 +-
docker/set-hubble-pd-password.sh | 80 ++++++++
docker/test-compose.sh | 126 +++++++++++-
.../static/conf/pd-application.yml.template | 13 +-
hugegraph-pd/README.md | 17 +-
hugegraph-pd/docs/api-reference.md | 58 ++----
hugegraph-pd/docs/configuration.md | 30 ++-
.../org/apache/hugegraph/pd/config/PDConfig.java | 38 +++-
.../hg-pd-dist/docker/docker-entrypoint.sh | 67 ++++++-
.../src/assembly/static/conf/application.yml | 21 +-
.../assembly/static/conf/application.yml.template | 13 +-
.../rest/interceptor/AuthenticationConfigurer.java | 10 +-
.../pd/rest/interceptor/RestAuthentication.java | 21 +-
.../pd/service/interceptor/Authentication.java | 56 +++++-
.../hugegraph/pd/util/grpc/GRpcServerConfig.java | 7 +-
.../src/main/resources/application.yml | 13 +-
.../pd/service/interceptor/AuthenticationTest.java | 220 +++++++++++++++++++++
.../java/org/apache/hugegraph/pd/BaseTest.java | 10 +-
.../apache/hugegraph/pd/rest/BaseServerTest.java | 14 ++
.../org/apache/hugegraph/pd/rest/RestApiTest.java | 100 +++++++++-
.../src/assembly/static/bin/wait-storage.sh | 81 +++++++-
.../hugegraph-dist/src/assembly/travis/start-pd.sh | 11 +-
.../assembly/travis/test-pd-docker-entrypoint.sh | 176 +++++++++++++++++
.../src/assembly/travis/test-pd-shipped-config.sh | 77 ++++++++
.../src/assembly/travis/test-wait-storage.sh | 119 ++++++++++-
hugegraph-store/README.md | 4 +-
hugegraph-store/docs/deployment-guide.md | 42 ++--
hugegraph-store/docs/integration-guide.md | 14 +-
hugegraph-store/docs/operations-guide.md | 14 +-
35 files changed, 1415 insertions(+), 190 deletions(-)
diff --git a/.github/workflows/pd-store-ci.yml
b/.github/workflows/pd-store-ci.yml
index 1a6825e7e..fce7168e0 100644
--- a/.github/workflows/pd-store-ci.yml
+++ b/.github/workflows/pd-store-ci.yml
@@ -134,6 +134,14 @@ jobs:
done
echo "can_run=true" >> "$GITHUB_OUTPUT"
+ - name: Run PD docker entrypoint secret override tests
+ run: |
+ $TRAVIS_DIR/test-pd-docker-entrypoint.sh
+
+ - name: Check every shipped PD config carries the REST hardening
+ run: |
+ $TRAVIS_DIR/test-pd-shipped-config.sh
+
- name: Run start-hugegraph-pd.sh foreground mode tests
if: steps.pd-preflight.outputs.can_run == 'true'
run: |
diff --git a/.gitignore b/.gitignore
index e1f546810..edfe89f7d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,6 +44,8 @@ build/
.env.test.local
.env.production.local
docker/.env
+# generated by docker/set-hubble-pd-password.sh, carries the PD REST secret
+docker/conf/hubble/*.local.properties*
*.orig
*.rej
diff --git a/docker/README.md b/docker/README.md
index 6fcfdee42..081fa36dc 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -39,33 +39,42 @@ contains a single quote or newline.
echo ".env already exists; edit it instead of overwriting it" >&2
exit 1
}
- printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" \
- 'replace-with-your-password' "${jwt_secret}" > .env
+ pd_secret="$(openssl rand -hex 24)"
+ printf
"HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\nHG_PD_AUTH_SECRET_KEY='%s'\n"
\
+ 'replace-with-your-password' "${jwt_secret}" "${pd_secret}" > .env
+ # Hubble reads the PD secret from a file, not from .env: generate the
untracked properties files the HStore topologies mount.
+ HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore
+ HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore-ha
)
```
-Do not commit `.env`. Keeping the same JWT secret preserves authentication
-tokens when containers are recreated. For authenticated topologies with
-multiple Server replicas, all replicas receive this same secret. The HA
-topology fails fast if authentication is enabled without this shared secret.
+Do not commit `.env` or `conf/hubble/*.local.properties`; both are in
`.gitignore`. Keeping the same JWT secret preserves authentication tokens when
containers are recreated. For authenticated topologies with multiple Server
replicas, all replicas receive this same secret. The HA topology fails fast if
authentication is enabled without this shared secret.
-A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and
-Hubble detects that mode automatically. Omitting the variable or setting it to
-an empty value disables authentication. Auth-off is only suitable for a
-trusted local environment; never expose it to a public or untrusted network.
-Hubble listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only
-behind an HTTPS reverse proxy and trusted network controls.
+A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and
Hubble detects that mode automatically. Omitting the variable or setting it to
an empty value disables authentication. Auth-off is only suitable for a trusted
local environment; never expose it to a public or untrusted network. Hubble
listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only behind an
HTTPS reverse proxy and trusted network controls.
-`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its
-first authenticated startup. Changing `.env` does not rotate an existing
-administrator password; use the HugeGraph user API for credential changes.
+`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its
first authenticated startup. Changing `.env` does not rotate an existing
administrator password; use the HugeGraph user API for credential changes.
-For the verification commands below, set the password in your current shell:
+For the verification commands below, load `.env` into your current shell and
set the password:
```bash
+set -a; . ./.env; set +a
ADMIN_PASSWORD='the-same-password-used-in-.env'
```
+The PD REST API (port 8620, HStore topologies only) requires HTTP Basic auth
(`hg:${HG_PD_AUTH_SECRET_KEY}`) for all endpoints except health/readiness
probes (`/v1/health`, `/v1/ready`). `HG_PD_AUTH_SECRET_KEY` is shared across
PD, Server (`bin/wait-storage.sh`), and Hubble
(`conf/hubble/*.local.properties` generated by `./set-hubble-pd-password.sh`).
+
+Verify registered stores:
+
+```bash
+curl -u "hg:${HG_PD_AUTH_SECRET_KEY}" http://localhost:8620/v1/stores
+```
+
+To regenerate Hubble configuration after modifying `.env`:
+
+```bash
+./set-hubble-pd-password.sh hstore # or hstore-ha
+```
+
### Standalone
This is the recommended quickstart.
@@ -181,8 +190,7 @@ Status:
docker compose -f docker-compose-3pd-3store-3server.yml ps
```
-Verify all published PD, Store, and Server endpoints, Server authentication,
-and Hubble:
+Verify all published PD, Store, and Server endpoints, Server authentication,
and Hubble:
```bash
for port in 8620 8621 8622; do
@@ -202,34 +210,11 @@ done
curl -fsS http://localhost:8088/about
```
-PD answers two unauthenticated probe endpoints. `/v1/health` is liveness only:
-it returns `200` as soon as the REST listener is up, even when the PD has no
-raft leader. `/v1/ready` returns `200` only while the PD sees a raft leader,
-and `503` otherwise. Each PD answers for itself: a single PD elects itself, and
-in a three-PD group the two that can reach each other elect a leader and turn
-ready, while a partitioned third keeps answering `503` until it sees that
-leader.
-
-The healthchecks in these files still gate on `/v1/health`, because
-`/v1/ready` ships from the next release onwards while the files run published
-images. Two things to know before pointing them at readiness:
-
-- Match on the body, not the status code. As of 1.7.0 PD answers `200` with
- `{"status":-1,"error":"Unauthorized!"}` on every path its auth interceptor
- does not exclude, a path that does not exist included, so a status-only
- probe reads a PD too old to have `/v1/ready` as ready. The body match holds
- whichever status a refusal carries. Gate with
- `curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'` instead.
-- Pin `HUGEGRAPH_VERSION` to a release that carries the endpoint, or build the
- images from source with `docker-compose.dev.yml`.
-
-The `HEALTHCHECK` baked into `hugegraph-pd/Dockerfile` is `/v1/health` as well.
-Both compose files override it, so it governs `docker run` and anything else
-inheriting the image probe, and those keep reading a PD without a quorum as
-healthy.
+PD answers two unauthenticated probe endpoints: `/v1/health` for liveness
(returns 200 once the REST listener is up, regardless of raft state), and
`/v1/ready` for readiness (returns 200 only when PD sees a raft leader, 503
otherwise).
-Open `http://localhost:8088` and sign in as `admin` with the password from
-`.env`.
+Compose healthchecks currently gate on `/v1/health` for compatibility with
published images. When targeting readiness on newer releases or source builds
(`docker-compose.dev.yml`), match on the response body (`curl -fsS
http://localhost:8620/v1/ready | grep -q '"ready":true'`).
+
+Open `http://localhost:8088` and sign in as `admin` with the password from
`.env`.
Stop containers while keeping them:
@@ -265,9 +250,7 @@ HUBBLE_IMAGE=hugegraph/hubble:latest \
docker compose -f docker-compose.yml up -d
```
-The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and
-Server `latest`; compatibility with versions older than 1.7 is not promised.
-Pin immutable image references when reproducibility is required.
+The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and
Server `latest`; compatibility with versions older than 1.7 is not promised.
Pin immutable image references when reproducibility is required.
### Server startup timeout
@@ -348,16 +331,15 @@ docker compose -f docker-compose-hstore.yml up -d --wait
### Hubble configuration
-The three small files under `conf/hubble/` contain only topology-specific
-discovery settings and container paths:
+The three small files under `conf/hubble/` contain only topology-specific
discovery settings, the PD REST credential (`operations.pd.username` and
`operations.pd.password`, which must match PD's `auth.secret-key`), and
container paths:
- `conf/hubble/standalone.properties` uses direct Server mode.
-- `conf/hubble/hstore.properties` uses one PD and one Store REST target.
-- `conf/hubble/hstore-ha.properties` uses all three PD peers and all three
- allowed Store REST targets.
+- `conf/hubble/hstore.properties.example` uses one PD and one Store REST
target.
+- `conf/hubble/hstore-ha.properties.example` uses all three PD peers and all
three allowed Store REST targets.
+
+The two HStore topologies mount the generated `*.local.properties` next to
these examples (see `set-hubble-pd-password.sh`), never the examples
themselves, so the PD secret stays out of tracked files.
-Hubble detects Server authentication through the Server API. Do not add an
-`auth.enabled` property or duplicate auth-on/auth-off configurations.
+Hubble detects Server authentication through the Server API. Do not add an
`auth.enabled` property or duplicate auth-on/auth-off configurations.
### Render and smoke checks
@@ -367,12 +349,9 @@ Render every topology with auth-on inputs before
submitting a change:
bash test-compose.sh render
```
-The HA render is mandatory even when local resources are insufficient to start
-its ten containers.
+The HA render is mandatory even when local resources are insufficient to start
its ten containers.
-Run focused auth-on smoke checks for standalone and minimal HStore with the
-corresponding `up -d --wait`, status, authentication, Hubble `/about`, and
-`down -v` commands from the Users section:
+Run focused auth-on smoke checks for standalone and minimal HStore with the
corresponding `up -d --wait`, status, authentication, Hubble `/about`, and
`down -v` commands from the Users section:
```bash
bash test-compose.sh smoke
@@ -384,6 +363,4 @@ Run the required local auth-off checks separately:
bash test-compose.sh smoke-auth-off
```
-The auth-off mode is intentionally excluded from the default CI matrix and must
-remain on a trusted local machine. Both smoke modes remove only the isolated
-Compose projects and volumes that they create.
+The auth-off mode is intentionally excluded from the default CI matrix and
must remain on a trusted local machine. Both smoke modes remove only the
isolated Compose projects and volumes that they create.
diff --git a/docker/conf/hubble/hstore-ha.properties
b/docker/conf/hubble/hstore-ha.properties.example
similarity index 76%
rename from docker/conf/hubble/hstore-ha.properties
rename to docker/conf/hubble/hstore-ha.properties.example
index a50c52c96..82c32248d 100644
--- a/docker/conf/hubble/hstore-ha.properties
+++ b/docker/conf/hubble/hstore-ha.properties.example
@@ -20,6 +20,12 @@ pd.enabled=true
server.direct_url=http://server0:8080
pd.peers=pd0:8686,pd1:8686,pd2:8686
pd.server=pd0:8620
+# PD REST credential. The password must equal PD's auth.secret-key, which has
+# no default. Do not edit this tracked example:
docker/set-hubble-pd-password.sh
+# generates the untracked .local.properties that Compose mounts, with the value
+# from HG_PD_AUTH_SECRET_KEY in .env.
+operations.pd.username=hubble
+operations.pd.password=
operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520]
upload_file.location=/hubble/data/upload-files
dashboard.address=
diff --git a/docker/conf/hubble/hstore.properties
b/docker/conf/hubble/hstore.properties.example
similarity index 75%
rename from docker/conf/hubble/hstore.properties
rename to docker/conf/hubble/hstore.properties.example
index 5de43c212..271ef9039 100644
--- a/docker/conf/hubble/hstore.properties
+++ b/docker/conf/hubble/hstore.properties.example
@@ -20,6 +20,12 @@ pd.enabled=true
server.direct_url=http://server:8080
pd.peers=pd:8686
pd.server=pd:8620
+# PD REST credential. The password must equal PD's auth.secret-key, which has
+# no default. Do not edit this tracked example:
docker/set-hubble-pd-password.sh
+# generates the untracked .local.properties that Compose mounts, with the value
+# from HG_PD_AUTH_SECRET_KEY in .env.
+operations.pd.username=hubble
+operations.pd.password=
operations.store.allowed_targets=[http://store:8520]
upload_file.location=/hubble/data/upload-files
dashboard.address=
diff --git a/docker/docker-compose-3pd-3store-3server.yml
b/docker/docker-compose-3pd-3store-3server.yml
index 9c066242e..999f2126f 100644
--- a/docker/docker-compose-3pd-3store-3server.yml
+++ b/docker/docker-compose-3pd-3store-3server.yml
@@ -73,6 +73,8 @@ x-server-environment: &server-environment
HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120}
HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-}
PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-}
+ # bin/wait-storage.sh polls the PD REST API, so it needs the same secret
+ PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in
.env; see docker/README.md}
x-server-common: &server-common
image: hugegraph/server:${HUGEGRAPH_VERSION:-latest}
@@ -110,6 +112,7 @@ services:
HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
HG_PD_DATA_PATH: /hugegraph-pd/pd_data
HG_PD_INITIAL_STORE_COUNT: 3
+ HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set
HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
ports: ["8620:8620", "8686:8686"]
volumes:
- hg-pd0-data:/hugegraph-pd/pd_data
@@ -128,6 +131,7 @@ services:
HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
HG_PD_DATA_PATH: /hugegraph-pd/pd_data
HG_PD_INITIAL_STORE_COUNT: 3
+ HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set
HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
ports: ["8621:8620", "8687:8686"]
volumes:
- hg-pd1-data:/hugegraph-pd/pd_data
@@ -146,6 +150,7 @@ services:
HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
HG_PD_DATA_PATH: /hugegraph-pd/pd_data
HG_PD_INITIAL_STORE_COUNT: 3
+ HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set
HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
ports: ["8622:8620", "8688:8686"]
volumes:
- hg-pd2-data:/hugegraph-pd/pd_data
@@ -239,7 +244,15 @@ services:
- "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088"
volumes:
- hubble-data:/hubble/data
- -
./conf/hubble/hstore-ha.properties:/hubble/conf/hugegraph-hubble.properties:ro
+ - type: bind
+ source: ./conf/hubble/hstore-ha.local.properties
+ target: /hubble/conf/hugegraph-hubble.properties
+ read_only: true
+ bind:
+ # The file is generated by set-hubble-pd-password.sh and is
+ # gitignored. Without this, Docker would create an empty
+ # directory at that path and Hubble would boot unconfigured.
+ create_host_path: false
healthcheck:
test:
- CMD-SHELL
diff --git a/docker/docker-compose-hstore.yml b/docker/docker-compose-hstore.yml
index f6f20bd89..0fcc3697f 100644
--- a/docker/docker-compose-hstore.yml
+++ b/docker/docker-compose-hstore.yml
@@ -39,6 +39,7 @@ services:
HG_PD_RAFT_PEERS_LIST: pd:8610
HG_PD_INITIAL_STORE_LIST: store:8500
HG_PD_DATA_PATH: /hugegraph-pd/pd_data
+ HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set
HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
ports:
- "8620:8620"
volumes:
@@ -97,6 +98,8 @@ services:
HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120}
HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-}
PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-}
+ # bin/wait-storage.sh polls the PD REST API, so it needs the same secret
+ PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in
.env; see docker/README.md}
ports:
- "8080:8080"
healthcheck:
@@ -120,7 +123,15 @@ services:
- "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088"
volumes:
- hubble-data:/hubble/data
- -
./conf/hubble/hstore.properties:/hubble/conf/hugegraph-hubble.properties:ro
+ - type: bind
+ source: ./conf/hubble/hstore.local.properties
+ target: /hubble/conf/hugegraph-hubble.properties
+ read_only: true
+ bind:
+ # The file is generated by set-hubble-pd-password.sh and is
+ # gitignored. Without this, Docker would create an empty
+ # directory at that path and Hubble would boot unconfigured.
+ create_host_path: false
healthcheck:
test:
- CMD-SHELL
diff --git a/docker/set-hubble-pd-password.sh b/docker/set-hubble-pd-password.sh
new file mode 100755
index 000000000..73d3f18a6
--- /dev/null
+++ b/docker/set-hubble-pd-password.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Generate the Hubble properties file a Compose topology mounts, with PD's
+# REST secret written in as operations.pd.password.
+#
+# usage: set-hubble-pd-password.sh <hstore|hstore-ha> [secret]
+#
+# Reads conf/hubble/<name>.properties.example (tracked) and writes
+# conf/hubble/<name>.local.properties (ignored by git), so the secret never
+# lands in a tracked file. The secret defaults to $HG_PD_AUTH_SECRET_KEY. The
+# value never goes through a sed replacement, where & # and backslash are
+# special, and backslashes are doubled for the .properties format. Run this
+# before `docker compose up`: the bind pins create_host_path: false, so a
+# missing target makes Compose refuse to start.
+#
+# The secret must be printable ASCII. PD compares it as UTF-8 bytes
+# (Authentication.verifySecret), while Hubble reads this file through
+# commons-configuration2, whose DEFAULT_ENCODING is ISO-8859-1, so a non-ASCII
+# secret decodes to different bytes on the two sides and gives a permanent 401
+# with no diagnostic anywhere. The README recipe generates hex, which is safe.
+set -euo pipefail
+
+name=${1:?usage: $0 <hstore|hstore-ha> [secret]}
+secret=${2:-${HG_PD_AUTH_SECRET_KEY:-}}
+dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/conf/hubble"
+example="${dir}/${name}.properties.example"
+out="${dir}/${name}.local.properties"
+
+[[ -f "$example" ]] || { echo "no such topology: ${name} (expected
${example})" >&2; exit 1; }
+[[ -n "$secret" ]] || { echo "secret is empty; load .env first (set -a; .
./.env; set +a)" >&2; exit 1; }
+case "$secret" in
+ *$'\n'*|*$'\r'*) echo "secret contains a line break, which a .properties
value cannot hold" >&2; exit 1 ;;
+esac
+# LC_ALL=C so the range is ordinal and the walk byte-wise: under the caller's
+# collation a non-ASCII character can sort inside \x20-\x7e and slip through.
+is_printable_ascii() {
+ local LC_ALL=C
+ case "$1" in
+ *[!$'\x20'-$'\x7e']*) return 1 ;;
+ esac
+}
+is_printable_ascii "$secret" || {
+ echo "secret must be printable ASCII: Hubble reads .properties as
ISO-8859-1, PD compares as UTF-8" >&2
+ exit 1
+}
+
+escaped=${secret//\\/\\\\}
+# java.util.Properties skips whitespace between the separator and the value, so
+# a secret that starts with a space would reach Hubble shortened while PD and
+# the Server kept the original. A backslash before it keeps that first byte.
+case "$escaped" in
+ [$' \t']*) escaped="\\${escaped}" ;;
+esac
+tmp=$(mktemp "${out}.XXXXXX")
+trap 'rm -f "$tmp"' EXIT
+{
+ printf '# Generated from %s by set-hubble-pd-password.sh; not tracked by
git.\n' "$(basename "$example")"
+ grep -v '^operations\.pd\.password=' "$example" || true
+ printf 'operations.pd.password=%s\n' "$escaped"
+} > "$tmp"
+# Hubble runs unprivileged and the mount is read-only, so the file must be
world-readable
+chmod 644 "$tmp"
+mv "$tmp" "$out"
+trap - EXIT
+echo "wrote ${out}"
diff --git a/docker/test-compose.sh b/docker/test-compose.sh
index 767303648..54418d413 100644
--- a/docker/test-compose.sh
+++ b/docker/test-compose.sh
@@ -18,9 +18,16 @@
set -Eeuo pipefail
+# Most assertions below are a bare `jq -e ... >/dev/null`, so `set -e` used to
+# end the run with no output at all and a CI log that said only "exit code 1".
+# Name the command and line that failed instead; the trap fires once per frame,
+# so the innermost assertion comes first and its callers follow.
+trap 'echo "FAILED at ${BASH_SOURCE[0]}:${LINENO}: ${BASH_COMMAND}" >&2' ERR
+
DOCKER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PASSWORD="ci-compose-password"
SECRET="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
+PD_SECRET="ci-compose-pd-secret"
VERSION="ci-version"
RENDER_HUBBLE_IMAGE="example.invalid/hugegraph/hubble:ci"
DATASOURCE="jdbc:h2:file:/hubble/data/hubble;DB_CLOSE_ON_EXIT=FALSE"
@@ -41,6 +48,7 @@ compose_auth() {
HUBBLE_IMAGE="${RENDER_HUBBLE_IMAGE}" \
HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \
HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \
+ HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \
docker compose "$@"
}
@@ -144,6 +152,36 @@ assert_hubble() {
' "${rendered}" >/dev/null
}
+# The two *.local.properties files are generated by set-hubble-pd-password.sh
+# and gitignored. Their binds must disable host-path creation, so a first
+# `docker compose up` that skipped the generator fails instead of getting an
+# empty directory mounted over Hubble's config.
+#
+# This reads the Compose file rather than the render on purpose. compose-go
+# tags ServiceVolumeBind.CreateHostPath `json:"create_host_path,omitempty"`
+# (checked in compose-go v2.7.0, the version inside the Compose 2.38.2 that
+# ubuntu-24.04 runners ship), and omitempty drops a false bool, so
+# `config --format json` emits the same `"bind": {}` for the long syntax with
+# create_host_path disabled and for the short syntax this pins away from. The
+# render simply cannot carry this contract on that version.
+assert_hubble_bind_pinned() {
+ local file="$1" properties="$2"
+ grep -Fq "source: ./conf/hubble/${properties}" "${file}" || {
+ echo "${file}: the ${properties} mount is not in long bind syntax" >&2
+ return 1
+ }
+ grep -Fq "create_host_path: false" "${file}" || {
+ echo "${file}: the ${properties} bind does not disable" \
+ "create_host_path" >&2
+ return 1
+ }
+ if grep -Fq "./conf/hubble/${properties}:/hubble/conf" "${file}"; then
+ echo "${file}: the ${properties} mount is back on the short bind" \
+ "syntax, which lets Docker create an empty directory there" >&2
+ return 1
+ fi
+}
+
assert_standalone() {
local rendered="$1"
assert_common "${rendered}" \
@@ -170,7 +208,9 @@ assert_hstore() {
assert_common "${rendered}" \
'["hubble","pd","server","store"]' \
'["hubble-data","pd-data","store-data"]'
- assert_hubble "${rendered}" "hstore.properties" server
+ assert_hubble "${rendered}" "hstore.local.properties" server
+ assert_hubble_bind_pinned "${DOCKER_DIR}/docker-compose-hstore.yml" \
+ "hstore.local.properties"
jq -e '
.services.pd.image == "hugegraph/pd:ci-version" and
.services.store.image == "hugegraph/store:ci-version" and
@@ -192,9 +232,9 @@ assert_hstore() {
.source == "store-data" and
.target == "/hugegraph-store/storage")
' "${rendered}" >/dev/null
- assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.properties" \
+ assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.local.properties" \
"pd.peers=pd:8686"
- assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.properties" \
+ assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.local.properties" \
"operations.store.allowed_targets=[http://store:8520]";
}
@@ -203,8 +243,11 @@ assert_ha() {
assert_common "${rendered}" \
'["hubble","pd0","pd1","pd2","server0","server1","server2","store0","store1","store2"]'
\
'["hg-pd0-data","hg-pd1-data","hg-pd2-data","hg-store0-data","hg-store1-data","hg-store2-data","hubble-data"]'
- assert_hubble "${rendered}" "hstore-ha.properties" \
+ assert_hubble "${rendered}" "hstore-ha.local.properties" \
server0 server1 server2
+ assert_hubble_bind_pinned \
+ "${DOCKER_DIR}/docker-compose-3pd-3store-3server.yml" \
+ "hstore-ha.local.properties"
jq -e '
all([.services.pd0, .services.pd1, .services.pd2][];
.image == "hugegraph/pd:ci-version" and
@@ -237,9 +280,9 @@ assert_ha() {
.healthcheck.retries == 30 and
.healthcheck.start_period == "1m0s")
' "${rendered}" >/dev/null
- assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.properties" \
+ assert_file_property
"${DOCKER_DIR}/conf/hubble/hstore-ha.local.properties" \
"pd.peers=pd0:8686,pd1:8686,pd2:8686"
- assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.properties" \
+ assert_file_property
"${DOCKER_DIR}/conf/hubble/hstore-ha.local.properties" \
"operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520]";
}
@@ -274,12 +317,78 @@ cleanup() {
if [[ -n "${ACTIVE_PROJECT}" ]]; then
compose_active down -v --remove-orphans >/dev/null 2>&1 || true
fi
+ restore_hubble_configs
[[ -z "${RENDER_DIR}" ]] || rm -rf "${RENDER_DIR}"
}
+# The HStore topologies mount conf/hubble/<name>.local.properties, which is
+# generated and untracked. Generate both with the CI secret before any render
+# or `up`; the binds pin create_host_path: false, so a missing file makes
+# Compose refuse to start. A developer's own local files are put back
+# afterwards.
+HUBBLE_BACKUP_DIR=""
+prepare_hubble_configs() {
+ local name
+ HUBBLE_BACKUP_DIR="$(mktemp -d)"
+ for name in hstore hstore-ha; do
+ local f="${DOCKER_DIR}/conf/hubble/${name}.local.properties"
+ [[ ! -f "${f}" ]] || cp -p "${f}"
"${HUBBLE_BACKUP_DIR}/${name}.local.properties"
+ "${DOCKER_DIR}/set-hubble-pd-password.sh" "${name}" "${PD_SECRET}"
>/dev/null
+ done
+}
+restore_hubble_configs() {
+ [[ -n "${HUBBLE_BACKUP_DIR}" ]] || return 0
+ local name
+ for name in hstore hstore-ha; do
+ local f="${DOCKER_DIR}/conf/hubble/${name}.local.properties"
+ if [[ -f "${HUBBLE_BACKUP_DIR}/${name}.local.properties" ]]; then
+ cp -p "${HUBBLE_BACKUP_DIR}/${name}.local.properties" "${f}"
+ else
+ rm -f "${f}"
+ fi
+ done
+ rm -rf "${HUBBLE_BACKUP_DIR}"
+ HUBBLE_BACKUP_DIR=""
+}
+
+# set-hubble-pd-password.sh must survive the characters a sed replacement
+# would mangle, keep the rest of the example, produce a world-readable file
+# for the read-only mount, and refuse an empty secret or unknown topology.
+hubble_password_helper_check() {
+ local f="${DOCKER_DIR}/conf/hubble/hstore.local.properties"
+ "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore 'a&b#c\d' >/dev/null
+ local line mode
+ line=$(grep '^operations\.pd\.password=' "${f}")
+ mode=$(stat -c '%a' "${f}" 2>/dev/null || stat -f '%Lp' "${f}")
+ [[ "${line}" == 'operations.pd.password=a&b#c\\d' ]] || {
+ echo "set-hubble-pd-password.sh mangled the secret: ${line}" >&2; exit
1; }
+ [[ "${mode}" == "644" ]] || {
+ echo "set-hubble-pd-password.sh wrote mode ${mode}, Hubble could not
read it" >&2; exit 1; }
+ grep -q '^pd.server=pd:8620$' "${f}" || {
+ echo "set-hubble-pd-password.sh dropped the example's other
properties" >&2; exit 1; }
+ ! "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore '' 2>/dev/null || {
+ echo "set-hubble-pd-password.sh accepted an empty secret" >&2; exit 1;
}
+ ! "${DOCKER_DIR}/set-hubble-pd-password.sh" nope 'x' 2>/dev/null || {
+ echo "set-hubble-pd-password.sh accepted an unknown topology" >&2;
exit 1; }
+ # Hubble decodes this file as ISO-8859-1 and PD compares UTF-8 bytes, so a
+ # non-ASCII secret would be a permanent 401 with nothing in any log.
+ ! "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore 'pässwörd' 2>/dev/null
|| {
+ echo "set-hubble-pd-password.sh accepted a non-ASCII secret" >&2; exit
1; }
+ # java.util.Properties drops whitespace after the separator, so a leading
+ # space has to survive as an escape or Hubble reads a shortened secret.
+ "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore ' lead' >/dev/null
+ line=$(grep '^operations\.pd\.password=' "${f}")
+ [[ "${line}" == 'operations.pd.password=\ lead' ]] || {
+ echo "set-hubble-pd-password.sh left a leading space unescaped:
${line}" >&2
+ exit 1; }
+ # put the CI value back for the render/smoke that follows
+ "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore "${PD_SECRET}" >/dev/null
+}
+
run_render() {
RENDER_DIR="$(mktemp -d)"
trap cleanup EXIT INT TERM
+ prepare_hubble_configs
render "${RENDER_DIR}/standalone.json" \
-f "${DOCKER_DIR}/docker-compose.yml"
render "${RENDER_DIR}/hstore.json" \
@@ -331,6 +440,8 @@ run_render() {
assert_startup_timeout "${RENDER_DIR}/hstore-empty.json" "" server
assert_startup_timeout "${RENDER_DIR}/ha-empty.json" "" \
server0 server1 server2
+
+ hubble_password_helper_check
echo "Compose render contracts passed"
}
@@ -340,6 +451,7 @@ compose_active() {
HUBBLE_IMAGE="${HUBBLE_IMAGE:-hugegraph/hubble:latest}" \
HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \
HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \
+ HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \
COMPOSE_PROGRESS=plain \
docker compose -p "${ACTIVE_PROJECT}" "${ACTIVE_FILES[@]}" "$@"
}
@@ -442,6 +554,7 @@ smoke() {
run_smoke() {
trap cleanup EXIT INT TERM
+ prepare_hubble_configs
smoke standalone false true "${DOCKER_DIR}/docker-compose.yml"
smoke hstore true true "${DOCKER_DIR}/docker-compose-hstore.yml"
}
@@ -449,6 +562,7 @@ run_smoke() {
run_smoke_auth_off() {
PASSWORD=""
trap cleanup EXIT INT TERM
+ prepare_hubble_configs
smoke standalone-anon false false "${DOCKER_DIR}/docker-compose.yml"
smoke hstore-anon true false "${DOCKER_DIR}/docker-compose-hstore.yml"
}
diff --git
a/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template
b/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template
index 87229aabc..55a1afcbd 100644
---
a/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template
+++
b/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template
@@ -27,7 +27,10 @@ management:
endpoints:
web:
exposure:
- include: "*"
+ # Allowlist, not "*": actuator has its own handler mapping, which
+ # the REST auth interceptor is not attached to, so anything exposed
+ # here is anonymous on this port. This list is what bounds it.
+ include: "health,metrics,prometheus"
logging:
config: 'file:./conf/log4j2.xml'
@@ -43,6 +46,14 @@ server:
# REST service port number
port : $REST_PORT$
+auth:
+ # Shared secret checked against the password of the Basic credential on every
+ # authenticated REST request. Required and deliberately empty: a secret in
the
+ # source tree is published to everyone. Generate one per deployment and give
+ # every REST client the same value. While empty, PD refuses every
+ # authenticated REST request.
+ secret-key:
+
pd:
# Storage path
data-path: ./pd_data
diff --git a/hugegraph-pd/README.md b/hugegraph-pd/README.md
index c940dae4c..505002556 100644
--- a/hugegraph-pd/README.md
+++ b/hugegraph-pd/README.md
@@ -100,6 +100,7 @@ Key configuration file: `conf/application.yml`
| `raft.address` | `127.0.0.1:8610` | Raft service address for this PD node |
| `raft.peers-list` | `127.0.0.1:8610` | Comma-separated list of all PD nodes
in the Raft cluster |
| `pd.data-path` | `./pd_data` | Directory for storing PD metadata and Raft
logs |
+| `auth.secret-key` | none (required) | Password required by the REST API with
an internal service name (`hg`, `store`, `hubble`, `vermeer`) via HTTP Basic
auth. No default is shipped; generate one per deployment and configure every
REST client (e.g. Hubble's `operations.pd.password`) with the same value |
#### Single-Node Example
@@ -157,11 +158,12 @@ For detailed configuration options and production tuning,
see [Configuration Gui
#### Docker Bridge Network Example
-When running PD in Docker with bridge networking (e.g.,
`docker/docker-compose-3pd-3store-3server.yml`), configuration is injected via
environment variables instead of editing `application.yml` directly. Container
hostnames are used instead of IP addresses:
+When running PD in Docker with bridge networking (e.g.,
`docker/docker-compose-3pd-3store-3server.yml`), configuration is injected via
environment variables instead of editing `application.yml` directly. Container
hostnames are used instead of IP addresses. `HG_PD_AUTH_SECRET_KEY` is required
by the image and must be the same value on every PD node and every PD REST
client; generate it once (`openssl rand -hex 24`) and keep it:
**pd0** container:
```bash
HG_PD_GRPC_HOST=pd0
+HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd0:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500
@@ -170,6 +172,7 @@ HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500
**pd1** container:
```bash
HG_PD_GRPC_HOST=pd1
+HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd1:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500
@@ -178,6 +181,7 @@ HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500
**pd2** container:
```bash
HG_PD_GRPC_HOST=pd2
+HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd2:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500
@@ -236,11 +240,15 @@ Build PD Docker image:
# From project root
docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:latest .
+# Generate the REST secret once and keep it: every PD REST client needs this
same value, and a new one silently breaks the clients already using the old
one. Store it somewhere durable rather than only in this shell.
+export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)"
+
# Run container
docker run -d \
-p 8620:8620 \
-p 8686:8686 \
-p 8610:8610 \
+ -e HG_PD_AUTH_SECRET_KEY="${HG_PD_AUTH_SECRET_KEY}" \
-e HG_PD_GRPC_HOST=<your-ip> \
-e HG_PD_RAFT_ADDRESS=<your-ip>:8610 \
-e HG_PD_RAFT_PEERS_LIST=<your-ip>:8610 \
@@ -280,6 +288,13 @@ docker/docker-compose-3pd-3store-3server.yml
- Ensure low latency (<5ms) between PD nodes for Raft consensus
- Open required ports: `8620` (REST), `8686` (gRPC), `8610` (Raft)
+### Security
+
+- Keep all three ports on a trusted network. The REST API on `8620` includes
management endpoints that mutate the cluster (peer changes, store removal, data
movement), and the gRPC and Raft ports carry no authentication.
+- REST requests need HTTP Basic auth: one of the internal service names (`hg`,
`store`, `hubble`, `vermeer`) with the `auth.secret-key` value as the password.
Health probes (`/v1/health`, `/v1/ready`, `/actuator/**`, `/v1/prom/targets/*`)
stay unauthenticated.
+- `auth.secret-key` has no shipped default, because a secret in the source
tree is published to everyone. Generate one per deployment (`openssl rand -hex
24`) and set it in the config file, or through `HG_PD_AUTH_SECRET_KEY`, which
the Docker image requires. Give every REST client the same value: the Server's
`bin/wait-storage.sh` reads `PD_AUTH_PASSWORD` (and `PD_AUTH_USER`, default
`store`), and Hubble reads `operations.pd.password`. A client left on a stale
secret gets 401, and `wait- [...]
+- An existing `conf/application.yml` carried over from an earlier release has
no `auth` block, and still carries `management.endpoints.web.exposure.include:
"*"`. PD then starts with an empty secret and refuses every authenticated REST
request, logging an error that names `auth.secret-key`, while `/actuator/env`,
`/actuator/configprops` and `/actuator/beans` stay anonymously readable on
`8620`. Before upgrading, add `auth.secret-key` and narrow that exposure to
`health,metrics,prometheus [...]
+
### Monitoring
PD exposes metrics via REST API at:
diff --git a/hugegraph-pd/docs/api-reference.md
b/hugegraph-pd/docs/api-reference.md
index 58b969886..f7c37be9d 100644
--- a/hugegraph-pd/docs/api-reference.md
+++ b/hugegraph-pd/docs/api-reference.md
@@ -760,6 +760,16 @@ for (Map.Entry<String, byte[]> entry : results.entrySet())
{
PD exposes a REST API for management and monitoring (default port: 8620).
+### Authentication
+
+Every endpoint below except the probes needs HTTP Basic auth: one of the
internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user, and
the `auth.secret-key` value from PD's `conf/application.yml` as the password. A
missing or wrong credential gets HTTP 401. The `curl` examples that follow omit
`-u` for readability; add it to every call except `/v1/health`, `/v1/ready`,
`/actuator/**` and `/v1/prom/targets/*`, which stay unauthenticated for probes.
+
+```bash
+curl -u hg:<secret> http://localhost:8620/v1/stores
+```
+
+Endpoints under `/v1` mutate the cluster (peer list changes, store removal,
partition balancing), so keep port 8620 on a trusted network regardless.
+
### Health Check
```bash
@@ -796,33 +806,13 @@ curl -i http://localhost:8620/v1/ready
}
```
-A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When
-the quorum is lost the PD keeps answering `/v1/health` with `200` but
-`/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the
-body carries no cluster addresses; the leader's address stays on `/v1/members`.
-
-The answer is served from state the raft callbacks maintain rather than from
-the raft node, so it stays prompt while an election is running and never waits
-on the node lock. `state` is therefore the last change raft announced. A PD
-reports `STATE_UNINITIALIZED` with `"ready": false` from process start until
-its first raft callback, which is the ordinary startup window before a quorum
-first forms, and jraft emits no callback for candidacy or leadership transfer,
-so a candidate reports `STATE_FOLLOWER` with `"ready": false`.
-
-Point Kubernetes readiness probes, `depends_on` healthchecks and any
-"wait for PD" script at `/v1/ready`; keep liveness probes on `/v1/health`
-so a PD that merely lost its leader is not restarted.
-
-Match on the body rather than on the status code alone. A PD that predates this
-endpoint does not reliably answer `404` for it: `RestAuthentication` refuses a
-request it does not exclude by writing an error envelope, and as of 1.7.0 it
-does so without setting a status, so an unknown path answers `200` with
-`{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore reads
-such a PD as ready. The body match holds whichever status a refusal carries: a
-shell gate should use
-`curl -fsS http://<pd-host>:8620/v1/ready | grep -q '"ready":true'`, and a
-Kubernetes `httpGet` probe should be paired with a PD image that carries the
-endpoint.
+A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When
the quorum is lost the PD keeps answering `/v1/health` with `200` but
`/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the
body carries no cluster addresses; the leader's address stays on `/v1/members`.
+
+The answer is served from state the raft callbacks maintain rather than from
the raft node, so it stays prompt while an election is running and never waits
on the node lock. `state` is therefore the last change raft announced. A PD
reports `STATE_UNINITIALIZED` with `"ready": false` from process start until
its first raft callback, which is the ordinary startup window before a quorum
first forms, and jraft emits no callback for candidacy or leadership transfer,
so a candidate reports `ST [...]
+
+Point Kubernetes readiness probes, `depends_on` healthchecks and any "wait for
PD" script at `/v1/ready`; keep liveness probes on `/v1/health` so a PD that
merely lost its leader is not restarted.
+
+Match on the body rather than on the status code alone. A PD that predates
this endpoint does not reliably answer `404` for it: `RestAuthentication`
refuses a request it does not exclude by writing an error envelope, and as of
1.7.0 it does so without setting a status, so an unknown path answers `200`
with `{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore
reads such a PD as ready. The body match holds whichever status a refusal
carries: a shell gate should use `curl [...]
### Metrics
@@ -856,19 +846,11 @@ Exported on `/actuator/prometheus` for alerting on quorum
loss:
| `hg_raft_has_leader` | `1` while this PD sees a leader (is inside a quorum),
`0` otherwise |
| `hg_raft_alive_peers` | Number of alive peers on the leader, itself
included; `NaN` elsewhere |
-`hg_raft_alive_peers` counts the peers the leader has heard from within the
-leader lease timeout, which jraft derives as 90% of the election timeout by
-default.
+`hg_raft_alive_peers` counts the peers the leader has heard from within the
leader lease timeout, which jraft derives as 90% of the election timeout by
default.
-A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when
-`hg_raft_has_leader == 0` on every member. Both are briefly true during a
-normal election, so alert on them with a `for:` clause longer than the
-election timeout rather than on the instantaneous value.
+A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when
`hg_raft_has_leader == 0` on every member. Both are briefly true during a
normal election, so alert on them with a `for:` clause longer than the election
timeout rather than on the instantaneous value.
-Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every
-node but the leader, and one `NaN` sample turns the result of `sum()` or
-`avg()` into `NaN` as well. Select the leader's series instead, for example
-`hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`.
+Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every
node but the leader, and one `NaN` sample turns the result of `sum()` or
`avg()` into `NaN` as well. Select the leader's series instead, for example
`hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`.
### Partition API
diff --git a/hugegraph-pd/docs/configuration.md
b/hugegraph-pd/docs/configuration.md
index e3ae4f6f2..4e60d8e8b 100644
--- a/hugegraph-pd/docs/configuration.md
+++ b/hugegraph-pd/docs/configuration.md
@@ -79,6 +79,32 @@ server:
- Metrics: `http://<host>:8620/actuator/metrics`
- Prometheus: `http://<host>:8620/actuator/prometheus`
+### REST Authentication Settings
+
+Every REST request except the probes below must carry HTTP Basic auth: one of
the internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user,
and the shared secret as the password. A missing or wrong credential gets HTTP
401. Unauthenticated paths: `/v1/health`, `/v1/ready`, `/actuator/**` and
`/v1/prom/targets/*`.
+
+```yaml
+auth:
+ secret-key: <a value you generate, e.g. `openssl rand -hex 24`>
+```
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `auth.secret-key` | String | none (required) | Password checked against the
Basic credential. There is no default: a secret shipped in the source tree
would be published to everyone. While it is empty PD refuses every
authenticated REST request and logs an error naming this parameter, and PD
refuses to start at all if it is set to the value that earlier revisions
carried as a placeholder. |
+
+Every REST client needs the same value: the Server's `bin/wait-storage.sh`
reads it from `PD_AUTH_PASSWORD`, Hubble from `operations.pd.password`, and the
Docker image takes `HG_PD_AUTH_SECRET_KEY`.
+
+```bash
+curl -u hg:<secret> http://<host>:8620/v1/stores
+```
+
+`-u` puts the secret in curl's process arguments, where any local account can
read it while the call runs, and PD REST is plain HTTP. On a shared host, or
across a network you do not control, keep the secret out of `argv` by reading
it from a file mode 0600:
+
+```bash
+printf 'user = "hg:%s"\n' "${PD_SECRET}" > pd.curlrc && chmod 600 pd.curlrc
+curl -K pd.curlrc http://<host>:8620/v1/stores
+```
+
### Raft Consensus Settings
Controls Raft consensus for PD cluster coordination.
@@ -253,13 +279,13 @@ management:
endpoints:
web:
exposure:
- include: "*" # Expose all actuator endpoints
+ include: "health,metrics,prometheus" # Allowlist; see note below
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `management.metrics.export.prometheus.enabled` | Boolean | `true` | Enable
Prometheus-compatible metrics at `/actuator/prometheus`. |
-| `management.endpoints.web.exposure.include` | String | `"*"` | Actuator
endpoints to expose. `"*"` = all, or specify comma-separated list (e.g.,
`"health,metrics"`). |
+| `management.endpoints.web.exposure.include` | String |
`"health,metrics,prometheus"` | Actuator endpoints to expose. Actuator is
served by its own handler mapping, which the REST authentication interceptor is
not attached to, so every endpoint listed here is reachable without a
credential on port 8620. This allowlist is what bounds which endpoints exist
there, so prefer it over `"*"`. The interceptor's `/actuator/**` exclusion
records the same intent but is not what makes these paths a [...]
## Deployment Scenarios
diff --git
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java
index acfc2ec29..56bd58b34 100644
---
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java
+++
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java
@@ -25,6 +25,7 @@ import java.util.regex.Pattern;
import org.apache.hugegraph.pd.ConfigService;
import org.apache.hugegraph.pd.IdService;
+import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
@@ -32,13 +33,23 @@ import org.springframework.stereotype.Component;
import lombok.Data;
import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
/**
* PD profile
*/
+@Slf4j
@Data
@Component
-public class PDConfig {
+public class PDConfig implements InitializingBean {
+
+ /**
+ * The secret that earlier revisions carried as the placeholder default for
+ * `auth.secret-key`. It is published in this repository, so a deployment
+ * still using it authenticates anyone who can read the source. Refuse to
+ * start rather than let a well-known string look like authentication.
+ */
+ private static final String PUBLISHED_SECRET_KEY =
"FXQXbJtbCLxODc6tGci732pkH1cyf8Qg";
// cluster ID
@Value("${pd.cluster_id:1}")
@@ -69,7 +80,11 @@ public class PDConfig {
@Autowired
private ThreadPoolGrpc threadPoolGrpc;
- @Value("${auth.secret-key: 'FXQXbJtbCLxODc6tGci732pkH1cyf8Qg'}")
+ // No default: Spring takes the text after the first ':' literally, so a
+ // quoted default would resolve to a value including the quotes and the
+ // leading space, and no client would ever match it. An absent key must
+ // yield "" so the REST interceptor can refuse every request and say why.
+ @Value("${auth.secret-key:}")
@ToString.Exclude
private String secretKey;
@@ -85,6 +100,25 @@ public class PDConfig {
private ConfigService configService;
private IdService idService;
+ @Override
+ public void afterPropertiesSet() {
+ if (PUBLISHED_SECRET_KEY.equals(this.secretKey)) {
+ throw new IllegalStateException(
+ "auth.secret-key is set to the value published in the
HugeGraph source " +
+ "tree, which authenticates anyone who can read it. Set a "
+
+ "deployment-specific secret in conf/application.yml, or
through the " +
+ "HG_PD_AUTH_SECRET_KEY environment variable for the Docker
image.");
+ }
+ // The shipped configs leave this empty on purpose. Say so in the boot
log:
+ // the REST interceptor also logs it, but only on the first refused
request,
+ // and /v1/health keeps answering 200 in the meantime.
+ if (this.secretKey == null || this.secretKey.isEmpty()) {
+ log.error("auth.secret-key is not configured, so every
authenticated REST " +
+ "request will be refused. Add it to conf/application.yml
(or set " +
+ "HG_PD_AUTH_SECRET_KEY) and give every REST client the
same value.");
+ }
+ }
+
public Map<String, String> getInitialStoreMap() {
if (initialStoreMap == null) {
initialStoreMap = new HashMap<>();
diff --git a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh
b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh
index 529936d06..77af2af89 100755
--- a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh
+++ b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh
@@ -26,10 +26,40 @@ require_env() {
fi
}
+# Escape a value for use inside a JSON string: backslash and quote, then every
+# remaining C0 control character as \uXXXX. Dropping only LF, as an earlier
+# version did, left CR and TAB to produce invalid JSON and a container that
+# failed before startup.
+#
+# LC_ALL=C is what makes the walk deterministic. eclipse-temurin:11-jre-jammy
+# sets LC_ALL=en_US.UTF-8, and under that locale `${#s}`/`${s:i:1}` iterate
+# characters rather than bytes and `<` sorts by collation rather than by code
+# point. Pinned to C, iteration is byte-wise, the comparison is ordinal, and
+# the bytes of a non-ASCII character (0x80 and up) pass through untouched,
+# which is still valid JSON. The escape branch then only sees single-byte
+# ASCII, so this pin is the only behavioral change.
json_escape() {
- local s="$1"
- s=${s//\\/\\\\}; s=${s//\"/\\\"}; s=${s//$'\n'/}
- printf "%s" "$s"
+ local LC_ALL=C
+ local s="$1" out="" i c
+ s=${s//\\/\\\\}
+ s=${s//\"/\\\"}
+ for (( i = 0; i < ${#s}; i++ )); do
+ c=${s:i:1}
+ case "$c" in
+ $'\n') out+='\n' ;;
+ $'\r') out+='\r' ;;
+ $'\t') out+='\t' ;;
+ $'\b') out+='\b' ;;
+ $'\f') out+='\f' ;;
+ *)
+ if [[ "$c" < $'\x20' || "$c" == $'\x7f' ]]; then
+ printf -v c '\\u%04x' "'$c"
+ fi
+ out+="$c"
+ ;;
+ esac
+ done
+ printf "%s" "$out"
}
migrate_env() {
@@ -51,14 +81,44 @@ require_env "HG_PD_GRPC_HOST"
require_env "HG_PD_RAFT_ADDRESS"
require_env "HG_PD_RAFT_PEERS_LIST"
require_env "HG_PD_INITIAL_STORE_LIST"
+# The REST API refuses every authenticated request without this, and the image
+# ships no default because a published secret is not a secret.
+require_env "HG_PD_AUTH_SECRET_KEY"
: "${HG_PD_GRPC_PORT:=8686}"
: "${HG_PD_REST_PORT:=8620}"
: "${HG_PD_DATA_PATH:=/hugegraph-pd/pd_data}"
: "${HG_PD_INITIAL_STORE_COUNT:=1}"
+# Actuator endpoints reachable without a credential. Hardened by default; an
+# operator who needs /actuator/info or /actuator/loggers from this image opts
+# in deliberately instead of losing the endpoint. "*" is refused: /actuator/env
+# returns the SPRING_APPLICATION_JSON entry below verbatim, secret included.
+: "${HG_PD_ACTUATOR_EXPOSURE:=health,metrics,prometheus}"
+if [[ "${HG_PD_ACTUATOR_EXPOSURE}" == *'*'* ]]; then
+ echo "ERROR: HG_PD_ACTUATOR_EXPOSURE must not be a wildcard;" \
+ "/actuator/env would publish the PD REST secret" >&2
+ exit 2
+fi
+
+# Secret for REST Basic authentication (auth.secret-key). Required above and
+# never logged.
+AUTH_JSON="\"auth\": { \"secret-key\": \"$(json_escape
"${HG_PD_AUTH_SECRET_KEY}")\" },"
+
+# The secret above lands in SPRING_APPLICATION_JSON, and actuator's /env
+# sanitizer keys off the property name: it redacts auth.secret-key but returns
+# the SPRING_APPLICATION_JSON environment entry verbatim, secret included. The
+# image's own conf/application.yml already narrows the exposure, but a
+# bind-mounted pre-1.8 config brings back include: "*". SPRING_APPLICATION_JSON
+# outranks the config file, so pin the allowlist here too, from the env-driven
+# default above rather than from a literal: every other PD setting in this
+# entrypoint is env-driven, and pinning a literal took away the operator's
+# only way to expose another endpoint from this image.
+MANAGEMENT_JSON="\"management\": { \"endpoints\": { \"web\": { \"exposure\": {
\"include\": \"$(json_escape "${HG_PD_ACTUATOR_EXPOSURE}")\" } } } },"
SPRING_APPLICATION_JSON="$(cat <<JSON
{
+ ${AUTH_JSON}
+ ${MANAGEMENT_JSON}
"grpc": { "host": "$(json_escape "${HG_PD_GRPC_HOST}")",
"port": "$(json_escape "${HG_PD_GRPC_PORT}")" },
"server": { "port": "$(json_escape "${HG_PD_REST_PORT}")" },
@@ -81,5 +141,6 @@ log " raft.peers-list=${HG_PD_RAFT_PEERS_LIST}"
log " pd.initial-store-list=${HG_PD_INITIAL_STORE_LIST}"
log " pd.initial-store-count=${HG_PD_INITIAL_STORE_COUNT}"
log " pd.data-path=${HG_PD_DATA_PATH}"
+log " management.endpoints.web.exposure.include=${HG_PD_ACTUATOR_EXPOSURE}"
./bin/start-hugegraph-pd.sh -d false -j "${JAVA_OPTS:-}"
diff --git a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml
b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml
index eb8fda452..ada979192 100644
--- a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml
+++ b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml
@@ -27,7 +27,10 @@ management:
endpoints:
web:
exposure:
- include: "*"
+ # Allowlist, not "*": actuator has its own handler mapping, which
+ # the REST auth interceptor is not attached to, so anything exposed
+ # here is anonymous on this port. This list is what bounds it.
+ include: "health,metrics,prometheus"
logging:
config: 'file:./conf/log4j2.xml'
@@ -44,6 +47,22 @@ server:
# REST service port number
port: 8620
+auth:
+ # Shared secret checked against the password of the Basic credential on every
+ # authenticated REST request. Clients send one of the internal service names
+ # (hg, store, hubble, vermeer) with this value as the password.
+ #
+ # REQUIRED, and deliberately empty: a secret shipped in the source tree is
+ # published to everyone, so there is no safe default to give you. Generate
+ # one per deployment, for example with `openssl rand -hex 24`, and set the
+ # same value on every client that calls the PD REST API (the Server's
+ # PD_AUTH_PASSWORD, Hubble's operations.pd.password).
+ #
+ # While this is empty PD refuses every authenticated REST request and logs
+ # an error naming this key. Do not expose port 8620 outside a trusted
+ # network even once it is set.
+ secret-key:
+
pd:
# Storage path
data-path: ./pd_data
diff --git
a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml.template
b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml.template
index 8b8f0d63c..43bdf99fc 100644
--- a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml.template
+++ b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml.template
@@ -29,7 +29,10 @@ management:
endpoints:
web:
exposure:
- include: "*"
+ # Allowlist, not "*": actuator has its own handler mapping, which
+ # the REST auth interceptor is not attached to, so anything exposed
+ # here is anonymous on this port. This list is what bounds it.
+ include: "health,metrics,prometheus"
grpc:
port: $GRPC_PORT$
@@ -42,6 +45,14 @@ grpc:
server:
port : $SERVER_PORT$
+auth:
+ # Shared secret checked against the password of the Basic credential on every
+ # authenticated REST request. Required and deliberately empty: a secret in
the
+ # source tree is published to everyone. Generate one per deployment and give
+ # every REST client the same value. While empty, PD refuses every
+ # authenticated REST request.
+ secret-key:
+
pd:
# Cluster ID: to distinguish different PD clusters
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
index d4b1e026d..e3da9b3a2 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
@@ -32,7 +32,15 @@ public class AuthenticationConfigurer implements
WebMvcConfigurer {
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(restAuthentication)
.addPathPatterns("/**")
- .excludePathPatterns("/actuator/*", "/v1/health", "/v1/ready",
+ // /actuator/** states the intent for nested probe paths such
as
+ // /actuator/metrics/{name}, but it is not what keeps them
open.
+ // Actuator is served by WebMvcEndpointHandlerMapping, which
only
+ // picks up MappedInterceptor beans; one added through this
+ // registry is never a bean, so it is attached to the MVC
handler
+ // mappings alone and never sees an actuator request either
way.
+ // What is reachable there is bounded by
+ // management.endpoints.web.exposure.include.
+ .excludePathPatterns("/actuator/**", "/v1/health", "/v1/ready",
"/v1/prom/targets/*");
}
}
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/RestAuthentication.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/RestAuthentication.java
index fbc129078..9d28e7d75 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/RestAuthentication.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/RestAuthentication.java
@@ -24,9 +24,9 @@ import java.util.function.Supplier;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
-import org.apache.hugegraph.pd.rest.API;
import org.apache.hugegraph.pd.service.interceptor.Authentication;
import org.springframework.lang.Nullable;
+import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.stereotype.Service;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.ModelAndView;
@@ -38,6 +38,7 @@ import lombok.extern.slf4j.Slf4j;
public class RestAuthentication extends Authentication implements
HandlerInterceptor {
private static final String TOKEN_KEY = "Pd-Token";
+ private static final String UNAUTHORIZED_BODY =
"{\"status\":-1,\"error\":\"Unauthorized\"}";
private static final Supplier<Boolean> DEFAULT_HANDLE = () -> true;
@Override
@@ -49,19 +50,31 @@ public class RestAuthentication extends Authentication
implements HandlerInterce
String authority = request.getHeader("Authorization");
if (authority == null) {
- throw new Exception("Unauthorized!");
+ throw new BadCredentialsException("Unauthorized!");
+ }
+ if (!authority.regionMatches(true, 0, "Basic ", 0, 6)) {
+ throw new BadCredentialsException("invalid basic
authentication info");
}
Function<String, Boolean> tokenCall = t -> {
response.addHeader(TOKEN_KEY, t);
return true;
};
- authority = authority.replace("Basic ", "");
+ authority = authority.substring(6);
return authenticate(authority, token, tokenCall, DEFAULT_HANDLE);
} catch (Exception e) {
+ response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
+ // RFC 7235 requires a challenge on a 401; without it clients that
+ // authenticate reactively never retry with credentials
+ response.setHeader("WWW-Authenticate", "Basic
realm=\"hugegraph-pd\"");
response.setContentType("application/json");
- response.getWriter().println(new API().toJSON(e));
+ // Constant body: the exception text named internal classes and
told an
+ // unauthenticated caller whether the name or the password was
wrong.
+ response.getWriter().println(UNAUTHORIZED_BODY);
response.getWriter().flush();
+ Throwable reason = e.getCause() != null ? e.getCause() : e;
+ log.debug("REST authentication refused for {} {}: {}",
request.getMethod(),
+ request.getRequestURI(), reason.getMessage());
return false;
}
}
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java
index 48bcf3868..4a8d73fa7 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java
@@ -18,22 +18,29 @@
package org.apache.hugegraph.pd.service.interceptor;
import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
import java.util.Base64;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
import java.util.function.Supplier;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hugegraph.pd.config.PDConfig;
+import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.stereotype.Component;
+import lombok.extern.slf4j.Slf4j;
+
/**
* Simple internal authentication component for PD service.
* <p>
- * <b>WARNING:</b> This class currently implements only basic internal
authentication
- * validation for internal modules (hg, store, hubble, vermeer). The
authentication mechanism
- * is designed for internal service-to-service communication only.
+ * <b>WARNING:</b> This class validates a Basic credential for internal modules
+ * (hg, store, hubble, vermeer): the service name must be one of the four and
the
+ * password must match the shared secret configured via `auth.secret-key`. The
+ * mechanism is designed for internal service-to-service communication only.
* </p>
*
* <p><b>Important SEC Considerations:</b></p>
@@ -57,10 +64,16 @@ import org.springframework.stereotype.Component;
* and regular security audits.
* </p>
*/
+@Slf4j
@Component
public class Authentication {
private static final Set<String> innerModules = Set.of("hg", "store",
"hubble", "vermeer");
+ private static final AtomicBoolean missingSecretLogged = new
AtomicBoolean();
+
+ @Autowired
+ private PDConfig pdConfig;
+
protected <T> T authenticate(String authority, String token,
Function<String, T> tokenCall,
Supplier<T> call) {
try {
@@ -70,26 +83,49 @@ public class Authentication {
}
byte[] bytes = authority.getBytes(StandardCharsets.UTF_8);
byte[] decode = Base64.getDecoder().decode(bytes);
- String info = new String(decode);
+ // RFC 7617: Basic credentials are UTF-8. Decoding with the
platform
+ // default would compare against a UTF-8 secret only when the host
+ // locale happens to agree.
+ String info = new String(decode, StandardCharsets.UTF_8);
int delim = info.indexOf(':');
if (delim == -1) {
throw new BadCredentialsException(invalidBasicInfo);
}
String name = info.substring(0, delim);
- // TODO: password validation is skipped — only service name is
checked against
- // innerModules. Full credential validation should be added as
part of the auth refactor.
- //String pwd = info.substring(delim + 1);
- if (innerModules.contains(name)) {
- return call.get();
- } else {
+ String pwd = info.substring(delim + 1);
+ if (!innerModules.contains(name)) {
throw new AccessDeniedException("invalid service name");
}
+ if (!verifySecret(pwd)) {
+ throw new BadCredentialsException("invalid credential");
+ }
+ return call.get();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
+ /**
+ * Compare the password of the Basic credential with the shared secret
+ * configured via `auth.secret-key`. A missing or empty secret refuses
every
+ * request instead of falling back to name-only authentication.
+ */
+ private boolean verifySecret(String pwd) {
+ String secret = this.pdConfig == null ? null :
this.pdConfig.getSecretKey();
+ if (StringUtils.isEmpty(secret)) {
+ // Logged once: this path is reachable by unauthenticated callers
+ if (missingSecretLogged.compareAndSet(false, true)) {
+ log.error("auth.secret-key is not configured, so every
authenticated REST " +
+ "request is refused. Add it to conf/application.yml
(or set " +
+ "HG_PD_AUTH_SECRET_KEY) and give every REST client
the same value.");
+ }
+ return false;
+ }
+ return MessageDigest.isEqual(pwd.getBytes(StandardCharsets.UTF_8),
+ secret.getBytes(StandardCharsets.UTF_8));
+ }
+
public static String getTokenKey(String name) {
return "PD/TOKEN/" + name;
}
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java
index 2b1103739..224d417cc 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java
@@ -41,7 +41,12 @@ public class GRpcServerConfig extends
GRpcServerBuilderConfigurer {
poolGrpc.getQueue()));
serverBuilder.maxInboundMessageSize(MAX_INBOUND_MESSAGE_SIZE);
// TODO: GrpcAuthentication is instantiated as a Spring bean but never
registered
- // here — add serverBuilder.intercept(grpcAuthentication) once auth is
refactored.
+ // here - add serverBuilder.intercept(grpcAuthentication) once auth is
refactored.
+ // It extends Authentication, which now also checks the Basic password
against
+ // auth.secret-key. Registering it therefore requires giving that
value to every
+ // gRPC client first: ServiceConstant.AUTHORITY (Server) and
DefaultPdProvider
+ // .authority (Store) are "" and "default" today, and hg-pd-cli sends
"".
+ // Otherwise no store can register once the interceptor is enabled.
}
}
diff --git a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml
b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml
index 5a03595f7..f8c1ecea8 100644
--- a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml
+++ b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml
@@ -27,7 +27,10 @@ management:
endpoints:
web:
exposure:
- include: "*"
+ # Allowlist, not "*": actuator has its own handler mapping, which
+ # the REST auth interceptor is not attached to, so anything exposed
+ # here is anonymous on this port. This list is what bounds it.
+ include: "health,metrics,prometheus"
grpc:
port: 8686
@@ -43,6 +46,14 @@ license:
server:
port: 8620
+auth:
+ # Shared secret checked against the password of the Basic credential on every
+ # authenticated REST request. Required and deliberately empty: a secret in
+ # the source tree is published to everyone. Set a deployment-specific value
+ # here (or via HG_PD_AUTH_SECRET_KEY) and give every REST client the same
+ # value. While empty, PD refuses every authenticated REST request.
+ secret-key:
+
pd:
# Periodically check whether the cluster is healthy at intervals, in seconds
patrol-interval: 300
diff --git
a/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java
b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java
new file mode 100644
index 000000000..8793642d3
--- /dev/null
+++
b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.pd.service.interceptor;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.lang.reflect.Field;
+import java.lang.reflect.Proxy;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hugegraph.pd.config.PDConfig;
+import org.apache.hugegraph.pd.rest.interceptor.RestAuthentication;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * In-process cover for the REST credential check. The suites that exercise it
+ * over HTTP talk to a PD in another JVM, so nothing here is covered by them.
+ */
+public class AuthenticationTest {
+
+ private static final String SECRET = "unit-test-secret";
+
+ private static Authentication authWithSecret(String secret) throws
Exception {
+ Authentication auth = new Authentication();
+ PDConfig config = new PDConfig();
+ config.setSecretKey(secret);
+ Field field = Authentication.class.getDeclaredField("pdConfig");
+ field.setAccessible(true);
+ field.set(auth, config);
+ return auth;
+ }
+
+ private static RestAuthentication restAuthWithSecret(String secret) throws
Exception {
+ RestAuthentication auth = new RestAuthentication();
+ PDConfig config = new PDConfig();
+ config.setSecretKey(secret);
+ Field field = Authentication.class.getDeclaredField("pdConfig");
+ field.setAccessible(true);
+ field.set(auth, config);
+ return auth;
+ }
+
+ private static boolean acceptsRest(RestAuthentication auth, String
authHeader) {
+ try {
+ HttpServletRequest req = (HttpServletRequest)
Proxy.newProxyInstance(
+ HttpServletRequest.class.getClassLoader(),
+ new Class<?>[]{HttpServletRequest.class},
+ (proxy, method, args) -> {
+ if ("getHeader".equals(method.getName())) {
+ return "Authorization".equalsIgnoreCase((String)
args[0]) ? authHeader : null;
+ }
+ if ("getMethod".equals(method.getName())) {
+ return "GET";
+ }
+ if ("getRequestURI".equals(method.getName())) {
+ return "/v1/stores";
+ }
+ return null;
+ });
+ StringWriter sw = new StringWriter();
+ PrintWriter pw = new PrintWriter(sw);
+ HttpServletResponse resp = (HttpServletResponse)
Proxy.newProxyInstance(
+ HttpServletResponse.class.getClassLoader(),
+ new Class<?>[]{HttpServletResponse.class},
+ (proxy, method, args) -> {
+ if ("getWriter".equals(method.getName())) {
+ return pw;
+ }
+ return null;
+ });
+ return auth.preHandle(req, resp, null);
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ private static String credential(String name, String pwd) {
+ return Base64.getEncoder().encodeToString(
+ (name + ":" + pwd).getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static boolean accepts(Authentication auth, String authority) {
+ try {
+ return auth.authenticate(authority, null, t -> Boolean.TRUE, () ->
Boolean.TRUE);
+ } catch (RuntimeException e) {
+ return false;
+ }
+ }
+
+ @Test
+ public void testEveryInnerModuleIsAcceptedWithTheSecret() throws Exception
{
+ Authentication auth = authWithSecret(SECRET);
+ for (String name : new String[]{"hg", "store", "hubble", "vermeer"}) {
+ Assert.assertTrue(name + " should be accepted with the right
secret",
+ accepts(auth, credential(name, SECRET)));
+ }
+ }
+
+ @Test
+ public void testPasswordIsActuallyChecked() throws Exception {
+ Authentication auth = authWithSecret(SECRET);
+ Assert.assertFalse("wrong password must be refused",
+ accepts(auth, credential("hg", "wrong-password")));
+ Assert.assertFalse("empty password must be refused",
+ accepts(auth, credential("hg", "")));
+ Assert.assertFalse("secret as the name must not help",
+ accepts(auth, credential(SECRET, SECRET)));
+ }
+
+ @Test
+ public void testUnknownServiceNameIsRefused() throws Exception {
+ Authentication auth = authWithSecret(SECRET);
+ Assert.assertFalse(accepts(auth, credential("nobody", SECRET)));
+ Assert.assertFalse(accepts(auth, credential("admin", SECRET)));
+ }
+
+ @Test
+ public void testMissingOrMalformedCredentialIsRefused() throws Exception {
+ Authentication auth = authWithSecret(SECRET);
+ Assert.assertFalse(accepts(auth, null));
+ Assert.assertFalse(accepts(auth, ""));
+ // no colon
+ Assert.assertFalse(accepts(auth, Base64.getEncoder().encodeToString(
+ "hg".getBytes(StandardCharsets.UTF_8))));
+ }
+
+ @Test
+ public void testUnconfiguredSecretRefusesEveryone() throws Exception {
+ for (String secret : new String[]{null, ""}) {
+ Authentication auth = authWithSecret(secret);
+ Assert.assertFalse("an unset secret must not fall back to a name
check",
+ accepts(auth, credential("hg", "")));
+ Assert.assertFalse(accepts(auth, credential("hg", SECRET)));
+ }
+ }
+
+ @Test
+ public void testNonAsciiSecretDoesNotDependOnTheDefaultCharset() throws
Exception {
+ String secret = "sécrèt-2026";
+ Authentication auth = authWithSecret(secret);
+ Assert.assertTrue(accepts(auth, credential("hg", secret)));
+ Assert.assertFalse(accepts(auth, credential("hg", "secret-2026")));
+ }
+
+ @Test
+ public void testPublishedSecretRefusesStartup() {
+ PDConfig config = new PDConfig();
+ config.setSecretKey("FXQXbJtbCLxODc6tGci732pkH1cyf8Qg");
+ Assert.assertThrows(IllegalStateException.class,
config::afterPropertiesSet);
+ }
+
+ @Test
+ public void testOwnSecretStarts() {
+ PDConfig config = new PDConfig();
+ config.setSecretKey(SECRET);
+ config.afterPropertiesSet();
+ }
+
+ @Test
+ public void testRestAuthenticationAcceptsCaseInsensitiveBasicScheme()
throws Exception {
+ RestAuthentication auth = restAuthWithSecret(SECRET);
+ for (String name : new String[]{"hg", "store", "hubble", "vermeer"}) {
+ String cred = credential(name, SECRET);
+ Assert.assertTrue("Basic with uppercase B should be accepted for "
+ name,
+ acceptsRest(auth, "Basic " + cred));
+ Assert.assertTrue("basic with all lowercase should be accepted for
" + name,
+ acceptsRest(auth, "basic " + cred));
+ Assert.assertTrue("BASIC with all uppercase should be accepted for
" + name,
+ acceptsRest(auth, "BASIC " + cred));
+ Assert.assertTrue("BaSiC with mixed case should be accepted for "
+ name,
+ acceptsRest(auth, "BaSiC " + cred));
+ }
+ }
+
+ @Test
+ public void testRestAuthenticationRefusesNonBasicOrMalformedSchemes()
throws Exception {
+ RestAuthentication auth = restAuthWithSecret(SECRET);
+ String cred = credential("hg", SECRET);
+ Assert.assertFalse("null Authorization header must be refused",
+ acceptsRest(auth, null));
+ Assert.assertFalse("empty Authorization header must be refused",
+ acceptsRest(auth, ""));
+ Assert.assertFalse("Basic with empty credentials must be refused",
+ acceptsRest(auth, "Basic "));
+ Assert.assertFalse("basic with empty credentials must be refused",
+ acceptsRest(auth, "basic "));
+ Assert.assertFalse("Bearer scheme must be refused",
+ acceptsRest(auth, "Bearer " + cred));
+ Assert.assertFalse("Digest scheme must be refused",
+ acceptsRest(auth, "Digest " + cred));
+ Assert.assertFalse("Basic without trailing space must be refused",
+ acceptsRest(auth, "Basic" + cred));
+ Assert.assertFalse("invalid credential with basic prefix must be
refused",
+ acceptsRest(auth, "basic " + credential("hg",
"wrong-secret")));
+ Assert.assertFalse("unknown service name with basic prefix must be
refused",
+ acceptsRest(auth, "basic " + credential("unknown",
SECRET)));
+ Assert.assertFalse("non-base64 token with basic prefix must be
refused",
+ acceptsRest(auth, "basic ???"));
+ }
+}
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java
index 0836120c7..d7df26d59 100644
---
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java
@@ -17,6 +17,9 @@
package org.apache.hugegraph.pd;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+
import org.apache.hugegraph.pd.client.PDConfig;
public class BaseTest {
@@ -24,9 +27,12 @@ public class BaseTest {
protected static String pdGrpcAddr = "127.0.0.1:8686";
protected static String pdRestAddr = "http://127.0.0.1:8620";;
protected static String user = "store";
- protected static String pwd = "";
+ // Must match the auth.secret-key that travis/start-pd.sh gives the PD
+ // under test; the shipped config carries no secret by design
+ protected static String pwd = "pd-ci-test-secret-not-for-production";
protected static String key = "Authorization";
- protected static String value = "Basic c3RvcmU6YWRtaW4=";
+ protected static String value = "Basic " +
Base64.getEncoder().encodeToString(
+ (user + ":" + pwd).getBytes(StandardCharsets.UTF_8));
protected PDConfig getPdConfig() {
return PDConfig.of(pdGrpcAddr).setAuthority(user, pwd);
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java
index 4aff85d1e..5c808b652 100644
---
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java
@@ -18,15 +18,29 @@
package org.apache.hugegraph.pd.rest;
import java.net.http.HttpClient;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
import org.junit.After;
import org.junit.BeforeClass;
public class BaseServerTest {
+ // Must match the auth.secret-key that travis/start-pd.sh gives the PD
+ // under test; the shipped config carries no secret by design
+ protected static final String SECRET =
"pd-ci-test-secret-not-for-production";
+ protected static final String AUTH_HEADER = "Authorization";
+ protected static final String VALID_AUTH = basicAuth("store", SECRET);
+
protected static HttpClient client;
protected static String pdRestAddr;
+ protected static String basicAuth(String name, String pwd) {
+ String credential = name + ":" + pwd;
+ return "Basic " + Base64.getEncoder()
+
.encodeToString(credential.getBytes(StandardCharsets.UTF_8));
+ }
+
@BeforeClass
public static void init() {
client = HttpClient.newHttpClient();
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
index da90f6f0f..c7c291746 100644
---
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
@@ -36,7 +36,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -55,7 +55,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/cluster";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -125,7 +125,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/members";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -139,7 +139,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/stores";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -153,7 +153,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/graphs";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -167,7 +167,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/highLevelPartitions";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -181,7 +181,7 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/partitions";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
@@ -194,11 +194,95 @@ public class RestApiTest extends BaseServerTest {
String url = pdRestAddr + "/v1/shards";
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url))
- .header("Authorization", "Basic
c3RvcmU6MTIz")
+ .header(AUTH_HEADER, VALID_AUTH)
.GET()
.build();
HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
JSONObject obj = new JSONObject(response.body());
assert obj.getInt("status") == 0;
}
+
+ @Test
+ public void testMissingCredentialGets401() throws URISyntaxException,
IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/v1/members";
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(new URI(url))
+ .GET()
+ .build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ assert response.statusCode() == 401;
+ }
+
+ @Test
+ public void testWrongPasswordGets401() throws URISyntaxException,
IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/v1/members";
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(new URI(url))
+ .header(AUTH_HEADER,
basicAuth("store", "wrong-password"))
+ .GET()
+ .build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ assert response.statusCode() == 401;
+ }
+
+ @Test
+ public void testEmptyPasswordGets401() throws URISyntaxException,
IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/v1/members";
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(new URI(url))
+ .header(AUTH_HEADER, basicAuth("hg",
""))
+ .GET()
+ .build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ assert response.statusCode() == 401;
+ }
+
+ private int statusWithoutCredential(String path) throws
URISyntaxException, IOException,
+
InterruptedException {
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(new URI(pdRestAddr + path))
+ .GET()
+ .build();
+ return client.send(request,
HttpResponse.BodyHandlers.ofString()).statusCode();
+ }
+
+ @Test
+ public void testProbePathsNeedNoCredential() throws URISyntaxException,
IOException,
+ InterruptedException {
+ // != 401, not == 200: /actuator/health answers 503 whenever any health
+ // indicator is DOWN (low disk space on a CI runner is the usual one)
+ // and /v1/health reflects cluster state, so == 200 would report a
+ // transient unhealthy PD as an authentication regression and send
+ // someone looking in this file. The claim here is only that these
+ // paths are reachable without a credential.
+ assert statusWithoutCredential("/v1/health") != 401;
+ assert statusWithoutCredential("/actuator/health") != 401;
+ // Nested actuator paths are probe surface too. They stay open because
+ // actuator has its own handler mapping that the auth interceptor is
not
+ // attached to, not because of the /actuator/** exclusion pattern.
+ assert statusWithoutCredential("/actuator/metrics/jvm.memory.used") ==
200;
+ }
+
+ @Test
+ public void testUnexposedActuatorEndpointIsClosed() throws
URISyntaxException, IOException,
+
InterruptedException {
+ // not in management.endpoints.web.exposure.include, so it must never
serve data
+ assert statusWithoutCredential("/actuator/env") != 200;
+ }
+
+ @Test
+ public void testUnknownServiceNameGets401() throws URISyntaxException,
IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/v1/members";
+ HttpRequest request = HttpRequest.newBuilder()
+ .uri(new URI(url))
+ .header(AUTH_HEADER,
basicAuth("nobody", SECRET))
+ .GET()
+ .build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ assert response.statusCode() == 401;
+ }
}
diff --git
a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh
b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh
index 93c3a19dd..168efe897 100755
--- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh
@@ -39,7 +39,30 @@ log() {
echo "[wait-storage] $1"
}
-PD_AUTH_ARGS="-u ${PD_AUTH_USER:-store}:${PD_AUTH_PASSWORD:-admin}"
+# PD REST credential. PD checks the password against its auth.secret-key and
+# ships no default, so this has to be provided by the deployment.
+# The value is deliberately kept out of the inner script's source text and out
+# of curl's argv: the inner shell reads it from the environment and hands it to
+# curl on stdin as a config file.
+PD_AUTH_USER="${PD_AUTH_USER:-store}"
+PD_AUTH_PASSWORD="${PD_AUTH_PASSWORD:-}"
+# curl -K reads one option per line and takes the value as a quoted string
+# whose only escapes are \\ \" \t \n \r \v. Backslash first, then the rest;
+# an unescaped line break would end the option early and send a truncated
+# credential (curl then warns that the remainder is an unknown option).
+escape_curlrc() {
+ local v=$1
+ v=${v//\\/\\\\}
+ v=${v//\"/\\\"}
+ v=${v//$'\n'/\\n}
+ v=${v//$'\r'/\\r}
+ v=${v//$'\t'/\\t}
+ v=${v//$'\v'/\\v}
+ printf '%s' "$v"
+}
+PD_AUTH_CURL_USER=$(escape_curlrc "${PD_AUTH_USER}")
+PD_AUTH_CURL_PASSWORD=$(escape_curlrc "${PD_AUTH_PASSWORD}")
+export PD_AUTH_CURL_USER PD_AUTH_CURL_PASSWORD
function key_exists {
local key=$1
@@ -93,33 +116,77 @@ if env | grep '^hugegraph\.' > /dev/null; then
export PD_REST_LIST
log "PD REST peers = $PD_REST_LIST"
+ # Only worth saying where PD is actually polled: topologies without
+ # pd.peers never send this credential anywhere.
+ if [ -z "${PD_AUTH_PASSWORD}" ]; then
+ log "WARN: PD_AUTH_PASSWORD is empty; PD will answer 401 unless
it runs without auth"
+ fi
log "Timeout = ${WAIT_STORAGE_TIMEOUT_S}s"
timeout "${WAIT_STORAGE_TIMEOUT_S}s" bash -c "
log() { echo '[wait-storage] '\"\$1\"; }
+ # curl stays out of the grep pipeline so its status code is
+ # readable: a 401 is a wrong secret, not a storage problem, and
+ # retrying it for 300s only hides that.
+ #
+ # 401s are counted, not flagged, so the abort is for the fleet
+ # and never for one peer. That distinction is real: PD serves
+ # /v1/stores well before stores finish registering, so the first
+ # pass of a rolling secret rotation can find one stale peer
+ # refusing while the healthy peers are merely storeless. A flag
+ # turned that into a dead Server. Returning 2 only when every
+ # peer polled refused keeps the fail-fast for a fleet-wide wrong
+ # secret, which still aborts on pass one instead of retrying for
+ # the full 300s.
check_any_pd_stores() {
+ refused=0
+ peers=0
for peer in \$(echo \"\$PD_REST_LIST\" | tr ',' ' '); do
- if curl ${PD_AUTH_ARGS} -f -s \
- --connect-timeout ${WAIT_STORAGE_PD_CONNECT_TIMEOUT_S} \
- --max-time ${WAIT_STORAGE_PD_MAX_TIMEOUT_S} \
- http://\${peer}/v1/stores 2>/dev/null | \
- grep -qi '\"state\"[[:space:]]*:[[:space:]]*\"Up\"'; then
+ peers=\$((peers + 1))
+ body=\$(printf 'user = \"%s:%s\"\n' \
+ \"\$PD_AUTH_CURL_USER\" \"\$PD_AUTH_CURL_PASSWORD\"
| \
+ curl -K - -s -w '\n%{http_code}' \
+ --connect-timeout
${WAIT_STORAGE_PD_CONNECT_TIMEOUT_S} \
+ --max-time ${WAIT_STORAGE_PD_MAX_TIMEOUT_S} \
+ \"http://\${peer}/v1/stores\"; 2>/dev/null)
+ code=\${body##*\$'\n'}
+ if [ \"\$code\" = 401 ]; then
+ log \"ERROR: PD at \${peer} refused the credential
(401):\" >&2
+ log ' PD_AUTH_PASSWORD must match PD
auth.secret-key' >&2
+ refused=\$((refused + 1))
+ continue
+ fi
+ if printf '%s' \"\$body\" | grep -qi
'\"state\"[[:space:]]*:[[:space:]]*\"Up\"'; then
echo \"\$peer\"
return 0
fi
done
+ [ \"\$peers\" -gt 0 ] && [ \"\$refused\" -eq \"\$peers\" ] &&
return 2
return 1
}
until PD_REST=\$(check_any_pd_stores); do
+ # Must stay the first statement in the loop: any command in
+ # front of it would overwrite \$? and turn the 401 abort back
+ # into a 300s retry.
+ rc=\$?
+ if [ \"\$rc\" -eq 2 ]; then exit 2; fi
log 'No Up store yet, retrying in 5s'
sleep 5
done
log \"Store registration check PASSED via \$PD_REST\"
log 'Storage backend is VIABLE'
- " || { echo "[wait-storage] ERROR: Timeout waiting for storage
backend"; exit 1; }
+ " || {
+ rc=$?
+ if [ "$rc" -eq 124 ]; then
+ echo "[wait-storage] ERROR: Timeout waiting for storage
backend"
+ else
+ echo "[wait-storage] ERROR: storage wait aborted, see the
message above"
+ fi
+ exit 1
+ }
else
log "No pd.peers configured, skipping storage wait"
diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh
b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh
index 0c137489e..94d4f82fe 100755
--- a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh
@@ -32,6 +32,15 @@ fi
PD_DIR=$HOME_DIR/hugegraph-pd/apache-hugegraph-pd-$VersionInBash
pushd $PD_DIR
-. bin/start-hugegraph-pd.sh
+# conf/application.yml ships auth.secret-key empty on purpose, so PD would
+# refuse every authenticated REST request. Supply a test-only secret; it must
+# match the value the PD test suites send. Keep it inside this subshell:
+# install-hstore.sh sources this script and start-store.sh in the same shell,
+# and an exported SPRING_APPLICATION_JSON would reach Store's Spring context
+# too.
+(
+ export
SPRING_APPLICATION_JSON='{"auth":{"secret-key":"pd-ci-test-secret-not-for-production"}}'
+ . bin/start-hugegraph-pd.sh
+)
sleep 10
popd
diff --git
a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh
new file mode 100755
index 000000000..ed39b6749
--- /dev/null
+++
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Checks that the PD Docker entrypoint turns HG_PD_AUTH_SECRET_KEY into valid
+# SPRING_APPLICATION_JSON, whatever the secret contains, that the value Spring
+# would read back is the secret that went in, and that the same document pins
+# the actuator exposure allowlist. The allowlist has to travel with the secret:
+# SPRING_APPLICATION_JSON outranks a bind-mounted conf/application.yml, and an
+# older one exposing every actuator endpoint would otherwise serve that secret
+# back from /actuator/env.
+
+set -euo pipefail
+
+ENTRYPOINT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." &&
pwd)/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh"
+TMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/pd-entrypoint-test.XXXXXX")
+trap 'rm -rf "${TMP_DIR}"' EXIT
+
+PASS=0
+FAIL=0
+
+[[ -f "${ENTRYPOINT}" ]] || { echo "entrypoint not found at ${ENTRYPOINT}"
>&2; exit 1; }
+command -v python3 >/dev/null || { echo "python3 required" >&2; exit 1; }
+
+mkdir -p "${TMP_DIR}/bin"
+cp "${ENTRYPOINT}" "${TMP_DIR}/docker-entrypoint.sh"
+# Stand in for the launcher: record the generated config instead of starting PD
+cat > "${TMP_DIR}/bin/start-hugegraph-pd.sh" <<'STUB'
+#!/usr/bin/env bash
+printf '%s' "${SPRING_APPLICATION_JSON}" > ./spring.json
+STUB
+chmod +x "${TMP_DIR}/bin/start-hugegraph-pd.sh"
"${TMP_DIR}/docker-entrypoint.sh"
+
+run_case() {
+ # run_case <name> <secret> [expected exposure] [extra NAME=VALUE ...]
+ local name="$1" secret="$2"
expected_exposure="${3:-health,metrics,prometheus}"
+ shift 2
+ (( $# == 0 )) || shift # drop the exposure argument; the rest is extra env
+ local out
+ if ! out=$(cd "${TMP_DIR}" && env \
+ "$@" \
+ HG_PD_GRPC_HOST=pd0 \
+ HG_PD_RAFT_ADDRESS=pd0:8610 \
+ HG_PD_RAFT_PEERS_LIST=pd0:8610 \
+ HG_PD_INITIAL_STORE_LIST=store0:8500 \
+ HG_PD_AUTH_SECRET_KEY="${secret}" \
+ ./docker-entrypoint.sh 2>&1); then
+ echo " FAIL ${name}: entrypoint exited non-zero"
+ printf '%s\n' "${out}" | tail -3
+ FAIL=$((FAIL + 1))
+ return
+ fi
+
+ if ! SECRET="${secret}" EXPOSURE="${expected_exposure}" \
+ python3 - "${TMP_DIR}/spring.json" <<'PY'
+import json, os, sys
+with open(sys.argv[1], encoding="utf-8") as fh:
+ doc = json.load(fh)
+got = doc["auth"]["secret-key"]
+want = os.environ["SECRET"]
+if got != want:
+ print(" round-trip mismatch: %r != %r" % (got, want))
+ sys.exit(1)
+exposure = doc.get("management", {}).get("endpoints", {}).get("web", {})
+exposure = exposure.get("exposure", {}).get("include")
+if exposure != os.environ["EXPOSURE"]:
+ print(" actuator exposure is %r, expected %r"
+ % (exposure, os.environ["EXPOSURE"]))
+ sys.exit(1)
+PY
+ then
+ echo " FAIL ${name}: invalid JSON, secret did not round-trip, or
exposure is wrong"
+ FAIL=$((FAIL + 1))
+ return
+ fi
+ echo " PASS ${name}"
+ PASS=$((PASS + 1))
+}
+
+echo "PD docker-entrypoint secret override"
+run_case "plain secret" 'aVerySecretValue123'
+run_case "carriage return" "$(printf 'a\rb')"
+run_case "tab" "$(printf 'a\tb')"
+run_case "double quote" 'a"b'
+run_case "backslash" 'a\b'
+run_case "backslash and quote" 'a\"b'
+run_case "non-ascii" 'sécrèt-2026'
+run_case "spaces" 'two words'
+
+# json_escape walks the secret with ${#s} and ${s:i:1} and compares with `<`.
+# Both are locale-sensitive, and the base image (eclipse-temurin:11-jre-jammy)
+# sets LC_ALL=en_US.UTF-8, so the same secret has to survive a UTF-8 locale
+# exactly as it does under C. Pick a UTF-8 locale the host actually has:
+# an ungenerated one silently falls back to C and makes the case vacuous.
+# Captured rather than piped into grep: `grep -q` exits at the first match, and
+# under pipefail the SIGPIPE that gives `locale -a` fails the whole pipeline.
+AVAILABLE_LOCALES=$(locale -a 2>/dev/null || true)
+UTF8_LOCALE=""
+for candidate in C.UTF-8 en_US.UTF-8 en_US.utf8; do
+ if grep -Fqix -- "${candidate}" <<<"${AVAILABLE_LOCALES}"; then
+ UTF8_LOCALE="${candidate}"
+ break
+ fi
+done
+if [[ -n "${UTF8_LOCALE}" ]]; then
+ run_case "non-ascii under ${UTF8_LOCALE}" 'sécrèt-2026' \
+ 'health,metrics,prometheus' "LC_ALL=${UTF8_LOCALE}"
+ run_case "control chars under ${UTF8_LOCALE}" "$(printf 'a\rb\tc')" \
+ 'health,metrics,prometheus' "LC_ALL=${UTF8_LOCALE}"
+else
+ echo " SKIP UTF-8 locale cases: no UTF-8 locale on this host"
+fi
+
+# The allowlist is the default, not a pin: an operator who needs another
+# endpoint from this image opts in rather than losing the endpoint entirely.
+run_case "actuator exposure override" 'aVerySecretValue123' \
+ 'health,metrics,prometheus,loggers' \
+ 'HG_PD_ACTUATOR_EXPOSURE=health,metrics,prometheus,loggers'
+
+# ...but not all the way back to the hole this closed: /actuator/env returns
+# the SPRING_APPLICATION_JSON entry verbatim, secret included.
+for wildcard in '*' 'health,*'; do
+ if (cd "${TMP_DIR}" && env \
+ HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \
+ HG_PD_RAFT_PEERS_LIST=pd0:8610
HG_PD_INITIAL_STORE_LIST=store0:8500 \
+ HG_PD_AUTH_SECRET_KEY='aVerySecretValue123' \
+ HG_PD_ACTUATOR_EXPOSURE="${wildcard}" \
+ ./docker-entrypoint.sh >/dev/null 2>&1); then
+ echo " FAIL wildcard exposure '${wildcard}' was accepted"
+ FAIL=$((FAIL + 1))
+ else
+ echo " PASS wildcard exposure '${wildcard}' is refused"
+ PASS=$((PASS + 1))
+ fi
+done
+
+# The secret is required, and must never be echoed to the log
+if (cd "${TMP_DIR}" && env \
+ HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \
+ HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \
+ ./docker-entrypoint.sh >/dev/null 2>&1); then
+ echo " FAIL missing secret: entrypoint started without
HG_PD_AUTH_SECRET_KEY"
+ FAIL=$((FAIL + 1))
+else
+ echo " PASS missing secret is refused"
+ PASS=$((PASS + 1))
+fi
+
+log_out=$(cd "${TMP_DIR}" && env \
+ HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \
+ HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \
+ HG_PD_AUTH_SECRET_KEY='do-not-log-this-value' \
+ ./docker-entrypoint.sh 2>&1)
+if printf '%s' "${log_out}" | grep -q 'do-not-log-this-value'; then
+ echo " FAIL secret was written to the log"
+ FAIL=$((FAIL + 1))
+else
+ echo " PASS secret is not logged"
+ PASS=$((PASS + 1))
+fi
+
+echo "${PASS} passed, ${FAIL} failed"
+[[ "${FAIL}" -eq 0 ]]
diff --git
a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh
new file mode 100755
index 000000000..46521dd05
--- /dev/null
+++
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Every PD configuration that ships in an archive or in the jar must carry the
+# same REST hardening: no wildcard actuator exposure (that path is anonymous),
+# an auth.secret-key that is present and empty, and no copy of the secret that
+# earlier revisions published. A fix applied to one variant and not the others
+# is what this catches, so the list below covers the PD distribution, the
+# service jar, and the template the cluster test writes onto each PD node.
+
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." && pwd)"
+PUBLISHED_SECRET='FXQXbJtbCLxODc6tGci732pkH1cyf8Qg'
+# The allowlist these files must carry, spelled out. An exact comparison rather
+# than "no wildcard": a missing include:, a reordered or duplicated entry, and
+# an extra endpoint are all changes to what this port serves anonymously, and
+# each of them used to pass.
+EXPECTED_EXPOSURE='health,metrics,prometheus'
+FAIL=0
+
+check() {
+ local file="$1" rel="${1#"${ROOT}/"}" bad=0
+ [[ -f "$file" ]] || { echo " FAIL ${rel}: missing"; FAIL=1; return; }
+
+ # The actuator exposure specifically: a config that grows an unrelated
+ # include: above this block must not satisfy the check by accident.
+ local exposure
+ exposure=$(awk '/^[[:space:]]*exposure:/ {found = 1; next}
+ found && /^[[:space:]]*include:/ {
+ sub(/^[[:space:]]*include:[[:space:]]*/, "")
+ sub(/[[:space:]]+$/, "")
+ print; exit
+ }' "$file")
+ # YAML quoting is the file's business, not this contract's
+ exposure=${exposure#\"}; exposure=${exposure%\"}
+ exposure=${exposure#\'}; exposure=${exposure%\'}
+ if [[ "$exposure" != "${EXPECTED_EXPOSURE}" ]]; then
+ echo " FAIL ${rel}: actuator exposure must be exactly" \
+ "'${EXPECTED_EXPOSURE}', got '${exposure}'"; FAIL=1; bad=1
+ fi
+ if ! grep -qE '^[[:space:]]*secret-key:[[:space:]]*$' "$file"; then
+ echo " FAIL ${rel}: auth.secret-key must be present and empty";
FAIL=1; bad=1
+ fi
+ if grep -q "${PUBLISHED_SECRET}" "$file"; then
+ echo " FAIL ${rel}: contains the published secret"; FAIL=1; bad=1
+ fi
+ # Per file, not the global FAIL: once an earlier file has set that, every
+ # later file matches it again and a failing file reports itself ok. Kept as
+ # an if rather than a bare test-and-echo, which would be the function's
last
+ # command and return 1 on a failing file, aborting the loop under set -e.
+ if [[ "$bad" -eq 0 ]]; then
+ echo " ok ${rel}"
+ fi
+}
+
+echo "PD shipped configuration hardening"
+for f in
"${ROOT}"/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml* \
+
"${ROOT}"/hugegraph-pd/hg-pd-service/src/main/resources/application.yml \
+
"${ROOT}"/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template;
do
+ check "$f"
+done
+[[ "$FAIL" -eq 0 ]] && echo "all shipped PD configs pass" || { echo "shipped
PD config check failed"; exit 1; }
diff --git
a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh
index da6a008f0..6d7c4ce8d 100755
--- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh
+++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh
@@ -25,6 +25,7 @@ DIST_ROOT="${TMP_DIR}/dist"
MOCK_BIN="${TMP_DIR}/mock-bin"
CALL_LOG="${TMP_DIR}/curl-calls"
ARGS_LOG="${TMP_DIR}/curl-args"
+CONFIG_LOG="${TMP_DIR}/curl-config"
COUNT_FILE="${TMP_DIR}/store-call-count"
TIMEOUT_LOG="${TMP_DIR}/timeout-arg"
CASE_OUTPUT=""
@@ -54,8 +55,12 @@ assert_contract() {
! grep -q '/v1/health' "${CALL_LOG}" || \
fail "/v1/health must not gate readiness"
[[ -s "${ARGS_LOG}" ]] || fail "curl was not called"
- if grep -Fv -- '-u test-user:test-password' "${ARGS_LOG}" | grep -q .; then
- fail "authentication arguments were not preserved"
+ if grep -Fq -- 'test-password' "${ARGS_LOG}"; then
+ fail "credential leaked into curl argv"
+ fi
+ [[ -s "${CONFIG_LOG}" ]] || fail "curl was not given a credential config"
+ if grep -Fv -- 'user = "test-user:test-password"' "${CONFIG_LOG}" | grep
-q .; then
+ fail "authentication credential was not preserved"
fi
if grep -Fv -- '--connect-timeout 2' "${ARGS_LOG}" | grep -q .; then
fail "per-peer connect timeout was not preserved"
@@ -67,9 +72,10 @@ assert_contract() {
}
run_case() {
- local scenario="$1" peers="$2" abort_after="$3"
+ local scenario="$1" peers="$2" abort_after="$3"
password="${4:-test-password}"
: > "${CALL_LOG}"
: > "${ARGS_LOG}"
+ : > "${CONFIG_LOG}"
: > "${COUNT_FILE}"
: > "${TIMEOUT_LOG}"
: > "${DIST_ROOT}/conf/graphs/hugegraph.properties"
@@ -80,11 +86,12 @@ run_case() {
MOCK_ABORT_AFTER="${abort_after}" \
MOCK_CALL_LOG="${CALL_LOG}" \
MOCK_ARGS_LOG="${ARGS_LOG}" \
+ MOCK_CONFIG_LOG="${CONFIG_LOG}" \
MOCK_COUNT_FILE="${COUNT_FILE}" \
MOCK_TIMEOUT_LOG="${TIMEOUT_LOG}" \
HG_SERVER_PD_REST_ENDPOINT="${peers}" \
PD_AUTH_USER="test-user" \
- PD_AUTH_PASSWORD="test-password" \
+ PD_AUTH_PASSWORD="${password}" \
'hugegraph.backend=hstore' \
'hugegraph.pd.peers=config-only:8686' \
"${DIST_ROOT}/bin/wait-storage.sh" 2>&1)
@@ -132,6 +139,26 @@ url="${!#}"
printf '%s\n' "$*" >> "${MOCK_ARGS_LOG}"
printf '%s\n' "${url}" >> "${MOCK_CALL_LOG}"
+# The credential must arrive as a config file on stdin, never in argv
+for arg in "$@"; do
+ if [[ "${arg}" == "-K" ]]; then
+ cat >> "${MOCK_CONFIG_LOG}"
+ break
+ fi
+done
+
+# Honour -w like curl: append the write-out with %{http_code} substituted
+fmt=""
+prev=""
+for arg in "$@"; do
+ [[ "${prev}" == "-w" ]] && fmt="${arg}"
+ prev="${arg}"
+done
+respond() {
+ printf '%s\n' "$1"
+ [[ -z "${fmt}" ]] || printf '%s' "${fmt//\\n/$'\n'}" | sed
"s/%{http_code}/$2/"
+}
+
if [[ "${url}" == */v1/health ]]; then
printf '{}\n'
exit 0
@@ -141,9 +168,25 @@ count=$(cat "${MOCK_COUNT_FILE}" 2>/dev/null || true)
count=$((${count:-0} + 1))
printf '%s\n' "${count}" > "${MOCK_COUNT_FILE}"
-if [[ "${MOCK_SCENARIO}" == "pd1-up" && \
+if [[ "${MOCK_SCENARIO}" == "auth-401" ]]; then
+ respond '{"status":-1,"error":"Unauthorized"}' 401
+elif [[ "${MOCK_SCENARIO}" == "one-401" && \
+ "${url}" == "http://pd0:8620/v1/stores"; ]]; then
+ # One stale peer mid-rotation, or a pre-1.8 peer that took any password
+ respond '{"status":-1,"error":"Unauthorized"}' 401
+elif [[ "${MOCK_SCENARIO}" == "one-401" && \
+ "${url}" == "http://pd1:8620/v1/stores"; ]]; then
+ respond '{"stores":[{"state":"Up"}]}' 200
+elif [[ "${MOCK_SCENARIO}" == "one-401-pending" && \
+ "${url}" == "http://pd0:8620/v1/stores"; ]]; then
+ # Same stale peer, but caught before any store has finished registering
+ respond '{"status":-1,"error":"Unauthorized"}' 401
+elif [[ "${MOCK_SCENARIO}" == "one-401-pending" && \
+ "${url}" == "http://pd1:8620/v1/stores"; ]]; then
+ respond '{"stores":[{"state":"Pending"}]}' 200
+elif [[ "${MOCK_SCENARIO}" == "pd1-up" && \
"${url}" == "http://pd1:8620/v1/stores"; ]]; then
- printf '{"stores":[{"state":"Up"}]}\n'
+ respond '{"stores":[{"state":"Up"}]}' 200
elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \
"${url}" == "http://pd0:8620/v1/stores"; ]]; then
if [[ " $* " == *" --connect-timeout 2 "* && \
@@ -155,15 +198,15 @@ elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \
exit 28
elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \
"${url}" == "http://pd1:8620/v1/stores"; ]]; then
- printf '{"stores":[{"state":"Up"}]}\n'
+ respond '{"stores":[{"state":"Up"}]}' 200
elif [[ "${MOCK_SCENARIO}" == "retry" && "${count}" -eq 3 && \
"${url}" == "http://pd0:8620/v1/stores"; ]]; then
exit 7
elif [[ "${MOCK_SCENARIO}" == "retry" && "${count}" -eq 4 && \
"${url}" == "http://pd1:8620/v1/stores"; ]]; then
- printf '{"stores":[{"state":"Up"}]}\n'
+ respond '{"stores":[{"state":"Up"}]}' 200
else
- printf '{"stores":[]}\n'
+ respond '{"stores":[]}' 200
fi
EOF
@@ -210,4 +253,60 @@ assert_output "ERROR: Timeout waiting for storage backend"
assert_contract
echo " PASS all-unready timeout"
-echo "5 passed, 0 failed"
+# A secret with CR/LF must reach curl as one escaped config line, not two.
+run_case "pd1-up" "pd0:8620,pd1:8620" 6 "$(printf 'a\r\nb\\c"d')"
+assert_equal "line-break secret rc" "0" "${CASE_RC}"
+if grep -Fv -- 'user = "test-user:a\r\nb\\c\"d"' "${CONFIG_LOG}" | grep -q .;
then
+ fail "line break or quote in the secret was not escaped for curl -K"
+fi
+echo " PASS line break in secret"
+
+# A fleet-wide wrong secret is not a storage problem: finish the pass so every
+# refusal is named, then abort rather than retrying it for the full 300s.
+run_case "auth-401" "pd0:8620,pd1:8620" 9
+[[ "${CASE_RC}" -ne 0 ]] || fail "a 401 from every peer must abort"
+assert_output "refused the credential (401)"
+assert_equal "one pass, no retry after 401" "${TWO_CALLS}" "$(cat
"${CALL_LOG}")"
+[[ "${CASE_OUTPUT}" != *"Timeout waiting"* ]] || fail "401 was reported as a
timeout"
+echo " PASS 401 from every peer aborts without retry"
+
+# One refusing peer must not cost the Server: during a rolling secret rotation,
+# or against a pre-1.8 peer that answers 200 to any password, the next peer in
+# PD_REST_LIST accepts the same credential. Only a refusal from every peer that
+# answered is a fleet-wide wrong secret, so a lone 401 stays a retry.
+run_case "one-401" "pd0:8620,pd1:8620" 6
+assert_equal "one refusing peer rc" "0" "${CASE_RC}"
+assert_equal "kept going past the 401" "${TWO_CALLS}" "$(cat "${CALL_LOG}")"
+assert_output "refused the credential (401)"
+assert_output "Store registration check PASSED via pd1:8620"
+echo " PASS one refusing peer does not end the wait"
+
+# The same lone 401 caught before any store is Up. PD serves /v1/stores well
+# ahead of store registration, so the first pass of a rolling rotation sees the
+# stale peer refuse and the healthy peers still storeless. That must retry, not
+# abort: the store is on its way, and the accepting peer is right there.
+run_case "one-401-pending" "pd0:8620,pd1:8620" 4
+[[ "${CASE_RC}" -ne 0 ]] || fail "a storeless fleet must still fail closed"
+assert_equal "retried past a lone 401" "${FOUR_CALLS}" "$(cat "${CALL_LOG}")"
+assert_output "refused the credential (401)"
+assert_output "No Up store yet, retrying in 5s"
+assert_output "ERROR: Timeout waiting for storage backend"
+[[ "${CASE_OUTPUT}" != *"storage wait aborted"* ]] || \
+ fail "a lone 401 aborted the wait with no Up store anywhere"
+echo " PASS one refusing peer with no Up store retries"
+
+# The standalone RocksDB topology never reaches PD, so it must not warn about
+# a PD credential it will not send.
+: > "${DIST_ROOT}/conf/graphs/hugegraph.properties"
+CASE_OUTPUT=$(env \
+ PD_AUTH_PASSWORD="" \
+ 'hugegraph.backend=rocksdb' \
+ "${DIST_ROOT}/bin/wait-storage.sh" 2>&1)
+CASE_RC=$?
+assert_equal "no-PD topology rc" "0" "${CASE_RC}"
+assert_output "No pd.peers configured, skipping storage wait"
+[[ "${CASE_OUTPUT}" != *"PD_AUTH_PASSWORD is empty"* ]] || \
+ fail "warned about an unused PD credential with no pd.peers configured"
+echo " PASS no credential warning without pd.peers"
+
+echo "10 passed, 0 failed"
diff --git a/hugegraph-store/README.md b/hugegraph-store/README.md
index 4e6bc8aca..f12df4427 100644
--- a/hugegraph-store/README.md
+++ b/hugegraph-store/README.md
@@ -259,8 +259,8 @@ curl http://localhost:8520/v1/health
# Check logs
tail -f logs/hugegraph-store.log
-# Verify registration with PD (from PD node)
-curl http://localhost:8620/v1/stores
+# Verify registration with PD (from PD node). PD REST needs HTTP Basic auth:
an internal service name (hg, store, hubble, vermeer) and PD's auth.secret-key
value. Without it this returns 401, not the store list.
+curl -u hg:"${PD_SECRET}" http://localhost:8620/v1/stores
```
For production deployment, see [Deployment Guide](docs/deployment-guide.md)
and [Best Practices](docs/best-practices.md).
diff --git a/hugegraph-store/docs/deployment-guide.md
b/hugegraph-store/docs/deployment-guide.md
index 40c70b0e0..ca39ae509 100644
--- a/hugegraph-store/docs/deployment-guide.md
+++ b/hugegraph-store/docs/deployment-guide.md
@@ -2,6 +2,14 @@
This guide provides comprehensive instructions for deploying HugeGraph Store
in various environments, from development to production clusters.
+> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than
`/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP
Basic auth: one of the internal service names (`hg`, `store`, `hubble`,
`vermeer`) and PD's `auth.secret-key` value as the password. A call without it
gets HTTP 401 and a `{"status":-1,"error":"Unauthorized"}` body, not the
payloads shown below. Export the secret before following a step that uses
`${PD_SECRET}`:
+>
+> ```bash
+> read -rs PD_SECRET && export PD_SECRET
+> ```
+>
+> Store endpoints on port 8520 are unaffected. `-u` puts the secret in curl's
process arguments; on a shared host pass it in a `curl -K` file mode 0600
instead, as `hugegraph-pd/docs/configuration.md` shows.
+
## Table of Contents
- [Deployment Topologies](#deployment-topologies)
@@ -472,7 +480,7 @@ curl http://localhost:8620/actuator/health
```bash
# Check cluster members
-curl http://192.168.1.10:8620/v1/members
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/members
# Expected output:
{
@@ -586,7 +594,7 @@ curl http://localhost:8520/v1/health
```bash
# Query PD for registered stores
-curl http://192.168.1.10:8620/v1/stores
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores
# Expected output:
{
@@ -678,6 +686,10 @@ For a production-like 3-node distributed deployment, use
the compose file at `do
```bash
cd docker
+# The PD REST secret is required; the Compose file refuses to start without
it. Generate it once and keep it, every PD node and PD client needs the same
value (docker/README.md has the full .env recipe).
+export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)"
+# Hubble reads the secret from a generated, untracked properties file that the
Compose file mounts; create it before `up` or Hubble starts unconfigured.
+./set-hubble-pd-password.sh hstore-ha
HUGEGRAPH_VERSION=1.7.0 docker compose -f
docker-compose-3pd-3store-3server.yml up -d
```
@@ -695,8 +707,13 @@ environment:
HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 # maps to
pd.initial-store-list
HG_PD_DATA_PATH: /hugegraph-pd/pd_data # maps to pd.data-path
HG_PD_INITIAL_STORE_COUNT: 3 # maps to
pd.initial-store-count
+ HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?} # maps to
auth.secret-key; required
+ # optional; maps to management.endpoints.web.exposure.include
+ HG_PD_ACTUATOR_EXPOSURE: health,metrics,prometheus
```
+`HG_PD_ACTUATOR_EXPOSURE` is the only way to change the actuator allowlist in
this image: the entrypoint emits it in `SPRING_APPLICATION_JSON`, which
outranks a mounted `conf/application.yml`. Add an endpoint here to expose it,
for example `health,metrics,prometheus,loggers`. A value containing `*` is
refused, because every actuator endpoint is anonymous on port 8620 and
`/actuator/env` returns the `SPRING_APPLICATION_JSON` entry verbatim, PD's REST
secret included.
+
**Store environment variables** (per node):
```yaml
@@ -723,10 +740,7 @@ environment:
2. Store nodes start after all PD nodes are healthy
3. Server nodes start after all Store nodes are healthy
-`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1
does
-not wait for a raft quorum to form. PD also serves `/v1/ready`, which answers
-`200` only while the PD sees a raft leader; `docker/README.md` covers what
-pointing the healthchecks at it requires.
+`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1
does not wait for a raft quorum to form. PD also serves `/v1/ready`, which
answers `200` only while the PD sees a raft leader; `docker/README.md` covers
what pointing the healthchecks at it requires.
> **Note**: The deprecated env var names (`GRPC_HOST`, `RAFT_ADDRESS`,
> `RAFT_PEERS`, `PD_ADDRESS`, `BACKEND`, `PD_PEERS`) still work but log a
> warning. Use the `HG_*` prefixed names for new deployments.
@@ -870,28 +884,22 @@ curl -i http://192.168.1.10:8620/v1/ready
curl http://192.168.1.20:8520/v1/health
```
-> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker
-> examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or
-> images built from source before this check means anything. On `1.7.0` the PD
-> answers `200` with `{"status":-1,"error":"Unauthorized!"}` on any path its
-> auth interceptor does not exclude, `/v1/ready` included, so match on the body
-> rather than the status code. See
-> [docker/README.md](../../docker/README.md) for the details.
+> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker
examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or images
built from source before this check means anything. On `1.7.0` the PD answers
`200` with `{"status":-1,"error":"Unauthorized!"}` on any path its auth
interceptor does not exclude, `/v1/ready` included, so match on the body rather
than the status code. See [docker/README.md](../../docker/README.md) for the
details.
### Cluster Status
```bash
# PD cluster members
-curl http://192.168.1.10:8620/v1/members
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/members
# Registered stores
-curl http://192.168.1.10:8620/v1/stores
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores
# Partitions
-curl http://192.168.1.10:8620/v1/partitions
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions
# Graph list
-curl http://192.168.1.10:8620/v1/graphs
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/graphs
```
### Basic Operations Test
diff --git a/hugegraph-store/docs/integration-guide.md
b/hugegraph-store/docs/integration-guide.md
index f35669c69..13164872e 100644
--- a/hugegraph-store/docs/integration-guide.md
+++ b/hugegraph-store/docs/integration-guide.md
@@ -2,6 +2,14 @@
This guide explains how to integrate HugeGraph Store with HugeGraph Server,
use the client library, and migrate from other storage backends.
+> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than
`/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP
Basic auth: one of the internal service names (`hg`, `store`, `hubble`,
`vermeer`) and PD's `auth.secret-key` value as the password. A call without it
gets HTTP 401. Export the secret before following a procedure that uses
`${PD_SECRET}`:
+>
+> ```bash
+> read -rs PD_SECRET && export PD_SECRET
+> ```
+>
+> Store endpoints on port 8520 are unaffected.
+
## Table of Contents
- [Backend Configuration](#backend-configuration)
@@ -706,7 +714,7 @@ tail -f logs/hugegraph-server.log | grep PD
curl http://192.168.1.20:8520/v1/health
# Check partition distribution
-curl http://192.168.1.10:8620/v1/partitions
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions
# Check if queries are using indexes
# (Enable query logging in Server)
@@ -733,10 +741,10 @@ ERROR o.a.h.b.s.h.HstoreSession - Write operation failed:
Raft leader not found
tail -f logs/hugegraph-store.log | grep Raft
# Check partition leaders
-curl http://192.168.1.10:8620/v1/partitions | grep leader
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions | grep leader
# Check Store node states
-curl http://192.168.1.10:8620/v1/stores
+curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores
```
**Solutions**:
diff --git a/hugegraph-store/docs/operations-guide.md
b/hugegraph-store/docs/operations-guide.md
index f46b5559d..e94bbf155 100644
--- a/hugegraph-store/docs/operations-guide.md
+++ b/hugegraph-store/docs/operations-guide.md
@@ -2,6 +2,14 @@
This guide covers monitoring, troubleshooting, backup & recovery, and
operational procedures for HugeGraph Store in production.
+> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than
`/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP
Basic auth: one of the internal service names (`hg`, `store`, `hubble`,
`vermeer`) and PD's `auth.secret-key` value as the password. A call without it
gets HTTP 401, which for a mutating step such as `balanceLeaders` means the
step did nothing. Export the secret before following a procedure that uses
`${PD_SECRET}`:
+>
+> ```bash
+> read -rs PD_SECRET && export PD_SECRET
+> ```
+>
+> Store endpoints on port 8520 are unaffected. Some PD examples in this guide
still omit the credential; add `-u hg:"${PD_SECRET}"` when a call returns 401.
+
## Table of Contents
- [Monitoring and Metrics](#monitoring-and-metrics)
@@ -604,19 +612,19 @@ curl http://192.168.1.10:8620/v1/partitionsAndStatus
2. **Verify Registration**:
```bash
- curl http://192.168.1.10:8620/v1/stores
+ curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores
# New Store should appear
```
3. **Trigger Rebalancing** (optional):
```bash
- curl -X POST http://192.168.1.10:8620/v1/balanceLeaders
+ curl -u hg:"${PD_SECRET}" -X POST http://192.168.1.10:8620/v1/balanceLeaders
```
4. **Monitor Rebalancing**:
```bash
# Watch partition distribution
- watch -n 10 'curl http://192.168.1.10:8620/v1/partitionsAndStatus'
+ watch -n 10 'curl -u hg:"${PD_SECRET}"
http://192.168.1.10:8620/v1/partitionsAndStatus'
```
5. **Verify**: Wait for even distribution (may take hours)
