[
https://issues.apache.org/jira/browse/HDDS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17154:
------------------------------
Attachment: HDDS-17154.001.patch
> S3 Gateway HAProxy in compose environments exits at startup if s3g1..s3g3 are
> not resolvable yet
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-17154
> URL: https://issues.apache.org/jira/browse/HDDS-17154
> Project: Apache Ozone
> Issue Type: Bug
> Components: docker-compose, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-17154.001.patch
>
>
> h3. Problem
> The S3 Gateway HAProxy in the compose environments can exit at startup and
> never come back. The affected environments are {{ozone-ha}} and
> {{ozonesecure-ha}} via {{common/s3-haproxy.yaml}}, and
> {{upgrade/compose/ha}}; all three use {{common/s3-haproxy.cfg}}.
> The current {{haproxy:lts-alpine}} (3.4.7) refuses to start if a server
> address cannot be resolved:
> {code}
> [ALERT] (8) : [/usr/local/etc/haproxy/haproxy.cfg:44] : 'server
> servers/server1' : could not resolve address 's3g1'.
> ...
> [ALERT] (8) : Failed to initialize server(s) addr.
> [WARNING] (1) : Failed to load worker (8) exited with code 1 (Exit)
> {code}
> Docker Compose starts services in parallel. If HAProxy starts before
> {{s3g1}}-{{s3g3}} are registered in Docker's DNS, it exits.
> Nothing restarts it, so the {{s3g}} name it serves is gone too. The compose
> files set {{dns_search: .}}, so {{s3g}} then falls through to public DNS
> ({{s3g.org}}). Every S3 robot test hangs until its 5 minute test timeout:
> {code}
> Try to create a bucket using S3 API | FAIL |
> Test timeout 5 minutes exceeded.
> {code}
> This was hit consistently while running {{upgrade/compose/ha}} on a loaded
> 4-core machine. I did not check CI history for it. This is a different cause
> than HDDS-16881, which fixed a {{grep -q}} + {{pipefail}} bug in the
> readiness wait.
> h3. Fix
> Make HAProxy resolve the S3 Gateways at runtime through Docker's embedded
> DNS, instead of only at startup, in {{common/s3-haproxy.cfg}}:
> {code}
> resolvers docker
> nameserver dns 127.0.0.11:53
> hold valid 10s
> defaults
> default-server init-addr last,libc,none resolvers docker
> {code}
> With {{init-addr ... none}}, HAProxy starts even if a server cannot be
> resolved yet. That server is disabled until the {{docker}} resolver finds it.
> One file covers all three environments.
> h3. Testing done
> * *Minimal compose reproduction* ({{haproxy:lts-alpine}} 3.4.7 with this
> config file; {{s3g1}}-{{s3g3}} are busybox httpd on 9878/19878). HAProxy was
> started alone, the gateways started afterwards, then a client requested
> {{http://s3g:9878/}}.
> ** Original config: HAProxy {{exited exit=1}} before the gateways existed and
> stayed exited. The request failed with {{bad address 's3g:9878'}}.
> ** Fixed config: HAProxy kept {{running}}, logged {{could not resolve address
> 's3g3', disabling server}} at startup, and later routed the request
> ({{HTTP/1.1 404 Not Found}} from the backend).
> * *{{ozone-ha/test-haproxy-s3g.sh}}* with the fixed config: passed, 149
> tests, 149 passed, 0 failed (built distribution with the fixed
> {{common/s3-haproxy.cfg}}).
> * In the {{upgrade/compose/ha}} downgrade test, the S3 robot steps went from
> timing out to passing once HAProxy could no longer exit at startup. There I
> used a compose override with {{depends_on}} and {{restart: on-failure}}; this
> patch fixes it in the shared config instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
