On 23/01/2024 22:44, Zach Pearson wrote:
Hello,
We are using haproxy in a few different environments/functions and the
configurations are getting complicated to manage. We are already using
Jinja2 to generate environment specific configs but was wondering if
anyone has suggestions on tools or processes we can adopt to help us
manage our configs.
Not sure how much you need to handle, but here's how we do it.
Our config is still in the "hand-written is ok" range, but with enough
to be really awful via a bunch of "use_backend if" (8 frontends, 70
backends):
- put your path/domain+path => backend matches in 2 dedicated files to
be used as a HAProxy map
- use map_ matches to reference it and do most of the routing
- then keep frontends entirely about sanitizing inputs
- and service-specific config purely inside the relevant backends, which
will be their own file with jinja includes
Ends up looking something like that (for us):
# pathonly.map
## things you want to match no matter the domain
/favicon generic_favicon
...
# hostpath.map
## things you match with domain+path
domain1#/ backend1
domain1#/someroute/ backend2
...
domain2#/ backend3
...etc
# haproxy.cfg
global
...
{{ include "frontend1.j2" }}
{{ include "frontend2.j2" }}
...
{{ include "backend1.j2" }}
{{ include "backend2.j2" }}
# frontend_common.j2
acl backend_match var(txn.selected_backend) -m found
... initial sanitizing etc ...
# (if sanitizing step didn't select a backend already, eg IP ban
handling)
http-request set-var(txn.route_path) path,lower if !backend_match
http-request set-var(txn.route_dompath)
hdr(host),concat(\#,txn.route_path) if !backend_match
# try path-only match first
http-request set-var(txn.selected_backend)
var(txn.route_path),map_beg(/path/to/pathonly.map) if !backend_match
# otherwise try domain+path
http-request set-var(txn.selected_backend)
var(txn.route_dompath),map_beg(/path/do/dompath.map) if !backend_match
use_backend %[var(txn.selected_backend,generic_404)]
# frontend1.j2
frontend frontend1
bind ...
{{ include "frontend_common.j2" }}
One good result is that I was quite surprised by how many
frontends+backends we actually have. One bad result is that
includes+macros make finding the source of a bad config a real pain
sometimes.
Also it's not super friendly to people discovering HAProxy, nor is it
the most readable, but you already lost the readability fight after the
25th line of "use_backend foo-bar if { hdr(Host) -i foo.domain } {
path_beg -i /bar }" anyway.
However, if you're already past the humanly-manageable step (or have too
many humans needing to cooperate on proxy configs), I see no good
solution other than the service registry model, similar to how ingress
controllers are implemented using k8s APIs:
- a program calls an API of yours (ICs call k8s, you have to make/find
your own)
- which returns sets of frontend, routes, certs
- and their matching backends
- your program then updates HAProxy via the dataplane/socket API to match
Assuming upstream teams can then self-register their service instances
on that API, it's mostly all dealt with for you.
As far as I've seen in the open:
- on the registry API side, Consul was essentially designed for this
- on the API<>HAProxy sync side, the haproxytech ingress controller
implementation could probably give you ideas (or be partially repurposed
for non-k8s use, which I said I'd open a GH issue for ages ago, and then
forgot to do...)
As an aside, I imagine that this is roughly how cloud providers
implement their LBaaS/APIGateway solutions, as it's similar in spirit to
the cloud-init+metadata-server setup they nearly all use for VM management.
Tristan
