Hi,
HAProxy 1.7-dev5 was released on 2016/10/25. It added 65 new commits
after version 1.7-dev4.
Things have been calming down since last release, which is quite good.
Among the changes I've noticed when preparing this release, the following
ones caught my attention:
- minimum supported OpenBSD version increased from 3.0 to 5.7 for the
"openbsd" make target. This brings accept4().
- API change for 51degrees, if you use it on the development version you
will have to read the notes to know how to upgrade your lib to the
newest version (or switch back to 1.6-stable which preserved the
compatibility with your current version)
- (hopefully) much safer and better architected DNS response parser
- long-awaited "tcp-request session" rule-set (just reminds me that
I forgot to update the ROADMAP file to mark it done). In short it
will mostly be used with track-sc actions when the frontend uses
the PROXY protocol.
- "noreuseport" option on bind lines to disable SO_REUSEPORT. May
sometimes be helpful.
- logs now ignore the idle and handshake times by default so that
the request time really represents the time from the first char
to the complete request. Idle and handshake times are available
separately.
- upgraded peers protocol to 2.1 (backwards compatible) to support
initial synchronization with expiration dates. It will solve the
problem that fast-reloaders are seeing with keys that never expire.
- bounded-load consistent hashing : limits the load on the target
server by spreading excess requests to neighbours. Andrew, I just
noticed that you documented "balance-factor" instead of
"hash-balance-factor", care to send me an updated patch ? (don't
forget to move the block to maintain alphabetical order). Thanks.
- CLI keyword registration : does nothing for now but will make it
possible to spread the various keywords to their respective files
as we did with sample fetches and converters, instead of constantly
overloading dumpstats.c.
- on the CLI we can now change a server's address, port and check port
at run time. Some related work is still in progress.
- IP_BIND_ADDRESS_NO_PORT is used when setting the source address of
outgoing connections on systems where it's supported (Linux >= 4.2).
- ah and the systemd saga patches to fix the zombie listening processes
during fast reloads.
Now let's stop speaking about the past and speak about the future instead.
First we may still have a few pending bugs. I got a private report about
a few zombie processes across a reload without systemd where each process
used to have CLOSE_WAIT connections on the peers port, so we seem to still
have an issue there. Reports are more than welcome.
Second, I know that we still have a few devs in progress. Baptiste is
finishing some really interesting changes to allow to start with a down
server that doesn't resolve that can be enabled at run time. I think everyone
understand what this allows to do. We should be able to start creating farms
with pre-provisionned servers that are configured on the fly so that many
users don't have to reload the process all the time.
Third, I used to have in my "list to santa claus" some drafts about a
protocol making it possible to let haproxy communicate with external
components to retrieve some info. The idea started with the problems
caused by most ldap libs not working fine in event-driven systems
(often at least the connect() is blocking). The concept would be to
reuse the co-sockets that Thierry created for the Lua network
communication for external validation. We can imagine sample fetches,
converters, actions, various things working more or less like an RPC,
with a standardized protocol that servers can easily implement without
getting their fingers dirty inside haproxy. Sort of a light ICAP. And
the good thing is that Christopher who did all the filters joined us
and accepted the challenge to try to reuse his filter infrastructure
to get this done for this release. Yeah Chris, feel the pressure now :-)
We won't be making an airport control tower first but something extensible,
so the initial design is more important than the functional coverage. I
still intend to release by the end of this month *if possible*, and given
that the end of this month is early next week, it's likely that it may
slip a little bit. But "a little bit" doesn't mean "let's take this
opportunity to make the first implementation even better" but "let's
ensure the first version works well enough". Obviously I'll take more
care of a well-designed protocol than an exact release date, but for
me the days of 1.5-dev are long dead and I'd rather release without
it that release 2 months late.
Fourth, while integrating the various patches I received recently, I
noticed that we still have a lot of ugly stuff in the code. If some
people are interested in contributing some time and/or code without
knowing where to start, here are a few good candidates for which I'm
willing to take patches, and where there's little risk, so don't be
shy :
- cfgparse.c : we have many "warnif_misplaced_..." functions that
repeat the same code parts. They can be factored so that each
function does its own test and relies on the previous function
for the other tests ; Estimated difficulty: 1/5.
- cfgparse.c : there are several loops doing the same resolve
operations on trk_ctr for various rulesets. This could be
improved with a list of list heads referencing all these rulesets
(or maybe almost all, I'm not picky but we don't need to have 6
times the same code). I'm even fine with a function doing this
and being called 6 times with 6 different pointers. Difficulty: 2/5.
- proto_tcp.c: the tcp-request rules were historically stored in
proto_tcp.c which now serves mostly for low-level TCP stuff and
networking. The tcp rules have no business staying there and
should move to their own file (eg: tcp_rules.{c,h}). Difficulty: 3/5.
- stream.c : the stick counters pollute that file a lot. I think we
almost don't need them to reference a stream at all anymore given
that some of them work on the session. Let's have stkctr.{c,h} and
move all that stuff off of there. Difficulty: 3/5.
- the stats counters are really boring. We have them per proxy, then
per bind line, then per server. In fact we have exactly the same
counters for listeners and frontends, and exactly the same counters
for servers and backends. So I think it would be much better to
define maybe "struct fe_counters" and "struct be_counters", and
have both of them in a proxy, then each one reused as-is by
listeners and servers. Then all manipulation places can be
simplified. Difficulty: 4/5. Beware of the copy-paste danger!
- dumpstats.c: many parts can be moved out of this file by using
the recent keyword registration stuff. Difficulty: 4/5.
There are many other things that can be cleaned up in my opinion but
these are the ones I noticed. If you're interested in picking one of
these tasks, please announce it here on the list (eg by just replying
to this email quoting only the part you intend to work on) to avoid
disappointment if two persons do the same thing. If something is not
done it's not dramatic so we won't blame you for saying "I'll try to
do this" if you finally fail. So it's a great opportunity to start
somewhere.
Last, Lukas is helping trying to get the haproxy github account usable
for us. More on that once it's ready depending on what we manage to do.
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.7/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.7/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
- MINOR: cfgparse: few memory leaks fixes.
- MEDIUM: log: Decompose %Tq in %Th %Ti %TR
- CLEANUP: logs: remove unused log format field definitions
- BUILD/MAJOR:updated 51d Trie implementation to incorperate latest update to
51Degrees.c
- BUG/MAJOR: stream: properly mark the server address as unset on connect
retry
- CLEANUP: proto_http: Removing useless variable assignation
- CLEANUP: dumpstats: Removing useless variables allocation
- CLEANUP: dns: Removing usless variable & assignation
- BUG/MINOR: payload: fix SSLv2 version parser
- MINOR: cli: allow the semi-colon to be escaped on the CLI
- MINOR: cli: change a server health check port through the stats socket
- BUG/MINOR: Fix OSX compilation errors
- MAJOR: check: find out which port to use for health check at run time
- MINOR: server: introduction of 3 new server flags
- MINOR: new update_server_addr_port() function to change both server's ADDR
and service PORT
- MINOR: cli: ability to change a server's port
- CLEANUP/MINOR dns: comment do not follow up code update
- MINOR: chunk: new strncat function
- MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value
- MINOR: dns: new MAX values
- MINOR: dns: new macro to compute DNS header size
- MINOR: dns: new DNS structures to store received packets
- MEDIUM: dns: new DNS response parser
- MINOR: dns: query type change when last record is a CNAME
- MINOR: dns: proper domain name validation when receiving DNS response
- MINOR: dns: comments in types/dns.h about structures endianness
- BUG/MINOR: displayed PCRE version is running release
- MINOR: show Built with PCRE version
- MINOR: show Running on zlib version
- MEDIUM: make SO_REUSEPORT configurable
- MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
- BUG/MEDIUM: http/compression: Fix how chunked data are copied during the
HTTP body parsing
- BUG/MINOR: stats: report the correct conn_time in backend's html output
- BUG/MEDIUM: dns: don't randomly crash on out-of-memory
- MINOR: Add fe_req_rate sample fetch
- MEDIUM: peers: Fix a peer stick-tables synchronization issue.
- MEDIUM: cli: register CLI keywords with cli_register_kw()
- BUILD: Make use of accept4() on OpenBSD.
- MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
- DOC: fix missed entry for "set-{src,dst}{,-port}"
- BUG/MINOR: vars: use sess and not s->sess in action_store()
- BUG/MINOR: vars: make smp_fetch_var() more robust against misuses
- BUG/MINOR: vars: smp_fetch_var() doesn't depend on HTTP but on the session
- MINOR: stats: output dcon
- CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
- MINOR: counters: add new fields for denied_sess
- MEDIUM: tcp: add registration and processing of TCP L5 rules
- MINOR: stats: emit dses
- DOC: document tcp-request session
- MINOR: ssl: add debug traces
- BUILD/CLEANUP: ssl: Check BIO_reset() return code
- BUG/MINOR: ssl: Check malloc return code
- BUG/MINOR: ssl: prevent multiple entries for the same certificate
- BUG/MINOR: systemd: make the wrapper return a non-null status code on error
- BUG/MINOR: systemd: always restore signals before execve()
- BUG/MINOR: systemd: check return value of calloc()
- MINOR: systemd: report it when execve() fails
- BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or
failed
- MINOR: proxy: add 'served' field to proxy, equal to total of all servers'
- MINOR: backend: add hash-balance-factor option for hash-type consistent
- MINOR: server: compute a "cumulative weight" to allow chash balancing to
hit its target
- MEDIUM: server: Implement bounded-load hash algorithm
- SCRIPTS: make git-show-backports also dump a "git show" command
- MINOR: build: Allow linking to device-atlas library file
- MINOR: stats: Escape equals sign on socket dump
---
