> On Sat, 6 Oct 2007, Brian Dickson wrote:
>
>> Consider the following set-up:
>>
>> A single prefix is announced by a single ASN, for each of which there
>> is only one instance. (I.e. non-anycast.)
>>
>> The prefix is used solely for offering services that are front-ended
>> by a stateful load-balancer pair. There are two LB's for redundancy
>> reasons. The LB's participate in the IGP of the ASN, also for
>> redundancy.
>>
>> The ASN (call it X) has two upstream ASNs, call them A and B. Each ISP
>> is connected to a separate router, for redundancy.
>
> A B
> \ /
> X
>
> Ok.
>
>> Inbound traffic from upstream ASN A hits router X-A, and inbound
>> traffic from upstream B hits router X-B.
>
> Ax Bx
> \ /
> Xa--Xb
>
> Still with you.
>
>> Router X-A prefers LB-A, and router X-B prefers LB-B.
>
> Not sure what you mean by 'LB' Assuming you mean prefer 'Link B to
> destination in A', I'm with you.
LB means Load Balancer. (You assume wrong.)
LB-A is the LB more closely associated with Xa and ISP A.
LB-B is the LB more closely associated with Xb and ISP B.
Both LB-A and LB-B front-end a set of servers providing service on the
prefix, which for clarity, we will call S.
It does not matter much whether the servers front-ended by LB-A and LB-B
are the same servers, or two distinct sets. The operational behaviour
will be identical.
The diagram looks like:
Ax Bx
| |
Xa---Xb
| |
LBa--LBb
\ /
B{1..n} (backend) servers 1 through N
On Xa, the preferred path for S is -> LBa.
On Xb, the preferred path for S is -> LBb.
>> LB-A and LB-B are state-independent. The LB support for stateful
>> traffic (e.g. TCP) works only if all incoming packets for a particular
>> TCP session hit the same LB.
>>
>> This is *not* anycast to the world. Some *may* consider it to be
>> IGP-anycast.
>
> Since you haven't reused the same IP addresses anywhere in this example,
> you don't have bgp anycast. IGP anycast isn't known. There may or may
> not be anycast within an AS. You example doesn't show that.
The prefix S is advertised by both LBa and LBb. The resulting preferred
paths are as described, i.e. each is equal-cost, and the shortest path
to either is used. The closest of LBa or LBb is used.
>> It is an extremely common set-up, perhaps *the* most common
>> configuration for load balancers.
This statement applies specifically to the detailed description and
diagram above, certainly from the perspective of Xa and Xb, and everything
below it in the diagram. The specifics of A, B, and additional ISPs and
peering above, are variations on the theme, but don't affect any aspect
of the primary functional description of what happens on any two AS paths
that ultimately connect on different Xn routers.
> I'll grant that one ISP with 2 upstreams is very common.
>
> However, each AS has its own load balancer with a unique IP address for
> each load balancer. When loadbalancers like the LTM redirect to other
> sites, (as the LTM example shows) it uses http redirection. It does not
> use Anycast.
In the diagram, there is exactly one ASN with load balancers. That is X.
X includes the AS border routers Xa and Xb. All of LBa, LBb, and B1 to Bn
are inside the AS X.
The load balancers do not have unique IP addresses. They have the same
IP address, call it S1. No other IP addresses from S are in use.
AS A and AS B have the same IP for both load balancers, and will only send
packets to S1.
All packets from A sent to the single IP address S1, get sent to LBa.
All packets from B sent to the single IP address S1, get sent to LBb.
The AS paths are unique. Every instance of S is originated from X, and
there is only one instance of X on the Internet. This is not BGP anycast.
(BTW - the LTM supports lots of other kinds of load balancing than http.
The profile examples for TCP and UDP, as well as L4 Fast, do not do http
redirect, for instance.)
>> Anyone using PPLB between the two AS paths, the one containing A
>> and the one contain B, *will* absolutely have problems using TCP, as
>> in, it won't work except in unusual circumstances (one of the two
>> paths is withdrawn, for instance.)
>
> Absolutely wrong. In _every_ non-anycast case, _all_ packets are
> delivered to _unique_ IP addresses, which identify individual, _unique_
> hosts. There is not requirement that all packets take the same path.
Nope. This is an instance where it is not anycast (as you seem to define
it at least). The IP address identifies a service.
There is no requirement that all packets take the same specific hop-by-hop
path.
However, if all packets from a tcp stream do not take the same *as*-path,
then all bets are off.
And in this instance, if the packets are distributed between the two
AS paths ending in "_A_X" and "_B_X", tcp flows will break, absolutely.
> The _only_ way that subsequent packets can arrive at the wrong host is
> to use Anycast. If you haven't used Anycast, there is no problem no
> matter how many different paths are used.
This example contradicts this assertion, and does not do (bgp) anycast.
> PPLB won't harmed anything in the non-Anycast case. RFC 1812 indicates
> that one can load balance packets across multiple paths. PPLB and the
> LTM will work just fine.
You should read the above. The non-anycast example above shows that
PPLB will not work in this case, regardless of make/model of LB, and
regardless of the mechanism used by the LB (be it an LTM or other).
BTW, RFC 1812 says only that "load-splitting" may work in some instances,
and may not be suitable for others. It predates BGP anycast, and lists
"load-splitting" in the section of future development that should be
watched. The only subsection of route selection that further references
load-splitting is "IGP" route selection, where equal cost paths can
be kept for load splitting.
Extrapolating this to mean that RFC 1812 explicitly supports PPLB,
is reading way to much into the document.
It makes the general observation that it might work. Your experience,
and the above example, serve as useful counter-examples.
Here is another example, in case you are interested in example cases
where PPLB via the same ASN example (A and B upstreams for X), can break
TCP:
If the upstream routers are both connected by Ethernet, and the server
pool used for providing load-balanced services is directly attached to
a single switch, which hashes the choice of server based on MAC addresses
plus IP addresses and ports, then the PPLB will break TCP.
Path via A will have, at layer 2, MAC_Xa + IP foo + port bar
Path via B will have, at layer 2, MAC_Xb + IP foo + port bar
These will, as a general rule, hash differently, and be sent to different
servers, if there is more than one server.
No anycast is needed at all.
AS-path PPLB is just an utterly unreliable way for delivering TCP-based
services.
(The irony is, that TCP becomes unreliable, but UDP is reliable, when
PPLB across AS paths is used - considering tcp is meant to be reliable,
and udp basically unreliable.)
Lots of other PPLB uses are still perfectly valid, and will be compatible
with the above examples - just as they would with bgp anycast.
Brian
_______________________________________________
DNSOP mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/dnsop
