Siyao Meng created HDDS-17154:
---------------------------------
Summary: S3 Gateway HAProxy in compose environments exits at
startup if s3g1..s3g3 are not resolvable yet
Key: HDDS-17154
URL: https://issues.apache.org/jira/browse/HDDS-17154
Project: Apache Ozone
Issue Type: Bug
Components: docker-compose, test
Reporter: Siyao Meng
Assignee: Siyao Meng
Attachments: HDDS-17154.001.patch
h3. Problem
The S3 Gateway HAProxy in the compose environments can exit at startup and
never come back. The affected environments are {{ozone-ha}} and
{{ozonesecure-ha}} via {{common/s3-haproxy.yaml}}, and {{upgrade/compose/ha}};
all three use {{common/s3-haproxy.cfg}}.
The current {{haproxy:lts-alpine}} (3.4.7) refuses to start if a server address
cannot be resolved:
{code}
[ALERT] (8) : [/usr/local/etc/haproxy/haproxy.cfg:44] : 'server
servers/server1' : could not resolve address 's3g1'.
...
[ALERT] (8) : Failed to initialize server(s) addr.
[WARNING] (1) : Failed to load worker (8) exited with code 1 (Exit)
{code}
Docker Compose starts services in parallel. If HAProxy starts before
{{s3g1}}-{{s3g3}} are registered in Docker's DNS, it exits.
Nothing restarts it, so the {{s3g}} name it serves is gone too. The compose
files set {{dns_search: .}}, so {{s3g}} then falls through to public DNS
({{s3g.org}}). Every S3 robot test hangs until its 5 minute test timeout:
{code}
Try to create a bucket using S3 API | FAIL |
Test timeout 5 minutes exceeded.
{code}
This was hit consistently while running {{upgrade/compose/ha}} on a loaded
4-core machine. I did not check CI history for it. This is a different cause
than HDDS-16881, which fixed a {{grep -q}} + {{pipefail}} bug in the readiness
wait.
h3. Fix
Make HAProxy resolve the S3 Gateways at runtime through Docker's embedded DNS,
instead of only at startup, in {{common/s3-haproxy.cfg}}:
{code}
resolvers docker
nameserver dns 127.0.0.11:53
hold valid 10s
defaults
default-server init-addr last,libc,none resolvers docker
{code}
With {{init-addr ... none}}, HAProxy starts even if a server cannot be resolved
yet. That server is disabled until the {{docker}} resolver finds it. One file
covers all three environments.
h3. Testing done
* *Minimal compose reproduction* ({{haproxy:lts-alpine}} 3.4.7 with this config
file; {{s3g1}}-{{s3g3}} are busybox httpd on 9878/19878). HAProxy was started
alone, the gateways started afterwards, then a client requested
{{http://s3g:9878/}}.
** Original config: HAProxy {{exited exit=1}} before the gateways existed and
stayed exited. The request failed with {{bad address 's3g:9878'}}.
** Fixed config: HAProxy kept {{running}}, logged {{could not resolve address
's3g3', disabling server}} at startup, and later routed the request ({{HTTP/1.1
404 Not Found}} from the backend).
* *{{ozone-ha/test-haproxy-s3g.sh}}* with the fixed config: passed, 149 tests,
149 passed, 0 failed (built distribution with the fixed
{{common/s3-haproxy.cfg}}).
* In the {{upgrade/compose/ha}} downgrade test, the S3 robot steps went from
timing out to passing once HAProxy could no longer exit at startup. There I
used a compose override with {{depends_on}} and {{restart: on-failure}}; this
patch fixes it in the shared config instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
