Tom, Lucy, Osama,
This draft looks like it could become important, so I wanted to review
it comprehensively. Particularly given my experience contributing to the
design of Generic UDP Tunneling (GUT; draft-manner-tsvwg-gut-02 (expired
Jan 2011)), which is very similar - as the GUE draft acknowledges.
In preparation for this review:
* I re-read all the (very useful) tsvwg mailing list comments about GUT
* I had to read a couple of dozen references (to catch up on the last
few years in this area)
* it's required some pretty deep thought, which has led me to have to
rewrite parts of it multiple times;
I'm afraid my review is about as long as the GUE draft itself. I should
probably turn all this into a Internet draft, but it's email for now. So
I've split it into 3 parts in separate emails:
A) Technical Review of GUE 'As-is' <--- This email
B) Editorial Review
C) Redesign of parts of GUE
* Pls read the review of "GUE as-is" first (ptA), it hopefully gives
solid arguments for why some parts of GUE's design are problematic.
* I'm afraid that is rather an understatement - I think I have
undermined nearly every part of the wire protocol: the version field,
the C flag, the Hlen field, and the flag-based options. And I believe
the semantics of the one remaining part (the proto/ctype field) misses
an opportunity to be a lot more powerful.
* Nonetheless, these are only my opinions at this stage. Therefore I
have disciplined myself to refer out to PtC for all redesign ideas, so
PtA remains solely about GUE "as-is".
* I hope you will accept the review in the spirit intended -
constructive criticism to improve the final result, altho I appreciate
you were probably hoping GUE was nearly done.
* I'm not proprietorial about any of the ideas I give in the redesign -
they are offered for the WG to use as it chooses. I don't really want to
be working on encapsulation stuff myself, I just end up having to
because a) encap is fundamental to real networking and it's always not
quite done right which makes everything else hard; and b) encap is often
the best way to get ahead of middebox evolution.
I should add that:
* I don't generally follow nvo3 (or intarea) lists. So apologies if some
of my points are duplicates.
* Nonetheless, this means my review is a good test of whether the draft
is comprehensible to an outsider.
* After I wrote this, I read Adrian Farrel's RTG Dir QA review. I don't
think I have directly duplicated any of his comments. We were uneasy
about some of the same things, but I have tried to complement criticism
with alternative design proposals (ptC).
* I noticed Adrian encouraged you to get review from the transport area.
I'm on the transport area review team, but I haven't been asked to do an
"official" transport area review of GUE. Whatever, the problems I have
uncovered are wider - best categorised as transport, protocol design
(encapsulation and extensibility), ops and security.
*EXEC SUMMARY**(of ptA Technical)
*
I've split the tech review up into the following parts, and I've
highlighted here where there are particularly serious problems:
1/ Addressing Architecture
For IETF standardization, connection semantics will need to be the
rule, not the exception. I know the exception applies where GUE came
from - private DCs. However nvo3 and the IETF more generally has to cope
with multi-tenant, multi-admin, and therefore firewalls (and other
middlebox crud).
I also identify some cases where GUE cannot work that will need to be
documented (not show-stoppers).
2/ Wire Protocol
I'm afraid I have unearthed a number of apparently nitty, but
actually serious show-stoppers (IMO). E.g. GUEv1 precludes future
versions of IP and GUE extensibility only works while there are no
extensions (!).
Also, the semantics of the ctype/proto field precludes some ideas we had
in GUT, but without really giving a reason. Perhaps you just hadn't
realised some potential uses of GUE that we had in mind.
This could be stated as: "Please don't unnecessarily constrain your
protocol design solely to the use-case(s) you have in mind." This is as
much a problem with the IETF process, which by default tries to
constrain a new protocol to the scope of one WG, even when it could be
more powerful. I've heard suggestions that GUE ought to move from nvo3
to intarea?, tsvwg?, which may help, but I don't know which would be
better. We should also bear in mind that a more powerful protocol can
become a more powerful attack weapon in the wrong hands, so strong
security review is also important.
3/ State
Important, but absent from the draft.
4/ Operation
Numerous, but mostly minor problems. The more serious ones are:
* no way for tunnels in tunnels to know which options to copy to the
outer, and which not.
* The claim that "GUE permits encap of arbitrary IP protocols" is
only true until it encounters a protocol it doesn't know (!).
An improved checksum solution is also presented (in PtC), which can
ensure checksum coverage of all non-mutable parts of a GUE packet and
traverses middleboxes even if they do not support zero checksums, while
at the same time minimising extra processing by generally avoiding
duplicate coverage.
5/ Security
I am worried about the new security options in GUE. Because they are
introduced within a completely new extension framework they will
introduce a whole set of new security vulnerabilities, flaws and bugs.
The security community is stretched enough as it is having to cover what
we already have. So it is important to justify why existing security
building blocks are insufficient for GUE (IMO, the relevant motivation
sections in the GUE extensions draft are insufficient).
I also highlight some new points about firewall interactions.
6/ Implementation
Just my little rant about LRO
Finally there's one endemic editorial problem that has led to a large
number of technical flaws and oversights. Over and over, the differences
between the main two modes of usage go unstated and unresolved. There
are only two short sections that discuss the two modes separately:
* Section 5.1 Network tunnel encap (adds GUE+UDP+IP outside an existing
IP header)
* Section 5.2 Transport layer encap (adds GUE+UDP between an existing
transport and an existing IP header).
The majority of the draft is written in the mindset of network tunnel
encap, but without saying so. If the reader is keeping both modes in
mind, this makes the draft very hard to understand. But also, some
fundamental problems (with one mode in some cases and the other mode in
other cases) have been overlooked by not considering each mode
separately at each stage of the discussion.
*TABLE OF CONTENTS**
*
Yes, a ToC for an an email!
A/ TECHNICAL PROBLEMS/COMMENTS
1/ Addressing Architecture
1.1/ Inferring Connection Semantics: the rule not the exception
1.2/ A Firewall or NAT in front of both ends
1.3/ Multiple GUE servers (transport encap) not possible behind a NAT-PT
with one external IP
1.4/ Network decap and transport decap problematic on the same (IP)
interface
2/ Wire Protocol
2.1/ HLEN too small
2.2/ GUE versions
2.3/ No need to interpret the protocol field relative to IPv4
2.4/ No need to restrict interpretation of the protocol field
2.5/ Missed opportunity to liberalise interpretation of the protocol field
2.6/ Positioning GUE with respect to existing IPv6 extension headers
2.7/ Reliable delivery of control messages
2.8/ Extensibility of the flags and optional fields scheme: doesn't work
2.9/ Hard-coded option lengths do not scale
2.10/ Random access to options needs motivating
3/ State
3.1/ Per-connection state vs. stateless connections but per-tunnel state
3.2/ Transport encap with Connection Semantics: Flow state management
3.3/ Keepalives for middlebox flow state
4/ Operation
4.1/ Transport encap: to GUE or not to GUE?
4.2/ Hop limit / TTL processing
4.3/ Error messages
4.4/ Tunnels in tunnels
4.5/ SHOULD adjust MTU?
4.6/ Is orig-proto field necessary in the fragmentation option?
4.7/ Congestion Control: reductio ad absurdum
4.8/ Multicast outer -> Implosion on inner destination
4.9/ Deriving flow entropy from the inner is contrary to "GUE permits
encap of arbitrary IP protocols" claim
4.10/ Flow entropy from encrypted data could weaken the crypto?
4.11/ No need to constrain flow entropy distribution
4.12/ No need to constrain flow entropy interpretation
5/ Security
5.1/ Addresses that are both visible and hidden? Have your GUE and eat
it too?
5.2/ How can the Security option protect a UDP/GUE header from being
moved or removed?
5.3/ What happens when a port scan sends a datagram to port 6080?
5.4/ Firewalls will still block new/atypical protocols
5.5/ Transport Encap: Two Passes through a Local Firewall?
6/ Implementation
6.1/ Practical Large Receive Offload Requirements
*A/ TECHNICAL PROBLEMS**/COMMENTS
*
_*1/ ADDRESSING ARCHITECTURE*__*
*_
*1.1/ Inferring Connection **Semantics: the rule not the exception
*
The draft assumes that, as a general rule, the UDP dst. port of a GUE
packet will be fixed (6080) and that flow entropy will come from the
source port (see the two quoted sections below).
S. 5.11.1. Flow classification
" ... When a packet is encapsulated with
GUE, the source port in the outer UDP packet is set to a flow
entropy value ...
S.5.11.2 Flow entropy properties
The flow entropy is the value set in the UDP source port of a
GUE packet. Flow entropy in the UDP source port should adhere to
the following properties:
Nonetheless, the draft recognises there will be cases where "connection
semantics" have to be applied in order to traverse middleboxes such as
firewalls and NATs (but only mentioned in the relevant parts of 5.6.1 &
5.6.2 quoted below).
Such middleboxes generally only allow "ingress" UDP datagrams if they
look like responses to recent "egress" datagram(s). So there has to be a
concept of an "initiator" end of the GUE tunnel. Only once the initiator
end has sent an "egress" datagram with src:dst ports e:G (from ephemeral
port e to the GUE port G), then the GUE encap at the remote "responder"
end would be able to traverse the middlebox using "ingress" datagrams
with src:dst ports reversed (G:e).
S.5.6.1. Inferring connection semantics:
A middlebox may infer bidirectional connection semantics
[...] To operate in
this environment, a GUE tunnel must assume connected semantics [...]
The source port set in the UDP
header must be the destination port the peer would set for replies.
In this case the UDP source port for a tunnel would be a fixed value
and not set to be flow entropy as described insection 5.11
<https://tools.ietf.org/html/draft-ietf-nvo3-gue-04#section-5.11>.
The selection of whether to make the UDP source port fixed or set to
a flow entropy value for each packet sent should be configurable for
a tunnel.
S. 5.6.2. NAT
In
the case of stateful NAT, connection semantics must be applied to a
GUE tunnel as described insection 5.6.1
<https://tools.ietf.org/html/draft-ietf-nvo3-gue-04#section-5.6.1>.
[BTW, I suggest changing the final sentence of the first para in
S.5.6.1. (quoted above) to:
Therefore, in the ingress direction, the destination UDP port would
provide flow entropy, while the source port would take the fixed
value of 6080 (the converse of the case insection 5.11
<https://tools.ietf.org/html/draft-ietf-nvo3-gue-04#section-5.11>).
]
The text quoted from both sections 5.6.1 & 5.6.2 above implies
a) that the operator of tunnel endpoint(s) can somehow know whether
there are any middleboxes within the tunnel.
b) that applying connection semantics is feasible.
Connection semantics feasibility:
* transport encap: relatively easy - it was simple to implement
connection semantics in GUT (see code
<http://www.netlab.tkk.fi/%7Ejmanner/gut.html> or example in Figure 4 in
draft-manner-tsvwg-gut-02
<https://tools.ietf.org/html/draft-manner-tsvwg-gut-02#section-3.4>, or
see description later under A3.2/ "Transport encap with Connection
Semantics: Flow state management"). Nonetheless, without congestion
semantics, GUE/GUT is even simpler, because it can be stateless.
* network encap: harder (see separate email for my proposed design: C1/
"Stateless Connection Semantics", but until there's a working
implementation we have to allow for the possibility that it's not feasible).
Regarding the first question - whether middleboxes (such as firewalls)
exist on a path:
* most operators of tunnel endpoints don't know for sure, but they do
know that firewalls, etc. are very likely, so they would have to turn on
the "middleboxes exist" parameter.
* in one or two important (but private) data centres, the admin might
know that there are no firewalls (and certainly no NATs), so she can
turn off the "middleboxes exist" parameter. However, that is the
exception not the rule.
In summary, connection semantics are essential wherever there might be
middleboxes. This implies:
* transport encap: connection semantics are relatively simple, so why
not solely standardize this case? The few cases where the operator knows
for certain that there are no middleboxes don't need to use connection
semantics, but they are in private networks, so they shouldn't be the
primary use-case for standardization.
* network encap: Will connection semantics work? Two possibilities:
a) if no, the GUE network encap will be pretty useless, given nearly
all real networks contain firewalls, etc. There will be no point
standardizing the network encap just for a few special private networks
that have no middleboxes.
b) if yes, they will be needed in most real networks, so it should be
the default case that is standardized. Then the IETF has to ask, is
there any point standardizing a GUE network encap without connection
semantics, just for a few controlled environments where the operator
knows for sure that there are no middleboxes?
Corollary of all this: A packet is a "GUE packet" if either src or dst
port = 6080.
*1.2/ A Firewall or NAT in front of both ends**
*
Most firewalls / NATs only allow an incoming UDP datagram in response to
a recent outgoing datagram. If there there are two such middleboxes each
"protecting" a different endpoint of a GUE tunnel (network or transport
encap), then neither end can send an initial GUE datagram.
To operate in such an environment, GUE endpoints will need to support
STUN [RFC5389].
*1.3/ **Multiple GUE servers (transport encap) not possible behind a
NAT-PT with one external IP**
*
Two cases:
* For transport encap: every GUE server has to have its own public IP
address.
Reason: if a NAT-PT with one external IP address (A) sits in front of
multiple GUE servers, only one can be reached on the well-known GUE port
(6080). Because there will be only one address:port combination to
address packets to (A:6080). (Dan Wing pointed out this same problem
with GUT on the tsvwg ML
<https://www.ietf.org/mail-archive/web/tsvwg/current/msg09851.html>).
It's not a killer, but it is a limitation to applicability that has to
be understood and documented.
* With network encap: Non-issue.
*1.4/ Network decap and transport decap problematic on the same***(IP)*
interface**
*
A consequence of using the same well-known port for GUE transport and
network encap is that both decaps cannot be deployed at the same IP
address.
Thought experiment: This might work by implementing a combined
transport/network decap that checked whether there was another IP header
in the header chain and:
* if there was, removed the outer IP and the outer UDP+GUE+option headers
* if not, removed solely the outer UDP+GUE+option headers, but not the
outer IP.
However, there is nothing to say that a GUE transport encap should not
encapsulate a packet that has already been tunnelled in an IP outer
(e.g. IPsec AH or ESP). That is, the transport encap would insert a UDP
and GUE header between the outer IP and the inner IP, without adding
another IP outer.
It would be safer to use two different well-known ports for transport
and network encap. However, I think deploying transport and network
encap on the same IP is a corner case we just need to rule as
inadmissible. Nonetheless, a sys-admin would get weird behaviour if this
did happen, with lots of head-scratching before she realised what had
happened. I'm not sure how to mitigate this.
_*2/ WIRE PROTOCOL*__*
*_
*2.1/ HLEN too small*
S3.1
The 5-bit Hlen field (multiplied in 4B units making max header length
128B) worries me a lot.
Let's not make a similar mistake to when we limited TCP option space to
40B, which has caused enormous grief.
*2.2/ GUE versions*
S3.1
The hack in GUE v1 to compress out the GUE header for direct
encapsulation of IP (v4 or v6) seems neat, but it is also /extremely
dangerous/. If GUE becomes successful, it would prevent incremental
deployment of any new version of IP starting 0b10, 0b11 or 0b00. Because:
* S.5.4 says drop an unknown version field, so IP cannot be upgraded
independently from GUE code.
* A version of IP starting 0b00 would be mistaken for GUE.
The latter might sound unlikely, but bear in mind that:
* you don't know what ideas might come up in future for using multiple
versions of IP - the IP version field could become important.
* a future version of IP might wrap the version field, because 0x0-0x3
are no longer used (a version only has to be a unique tag, it doesn't
have to increase).
[Aside: If you prefer an equally dangerous hack (perhaps because you
don't believe there will ever be a version of IP beyond v6), you could
have reduced the Ver field to the first single bit by making GUEv0 the
one without a GUE header, and GUEv1 the one with. This would have given
more space for the Hlen field (see my concern in A2.1/ "HLEN too small"
above and my idea in a separate email to remove the C flag).]
In the separate email about redesign, I'll describe an alternative
approach that always fits the base GUE protocol into 4B, or even within
the 8B UDP header (see C6/ Wire Protocol; it comes from an idea to
develop GUT into what I called Gutless
<https://www.ietf.org/mail-archive/web/tsvwg/current/msg09854.html>,
back in Feb 2010).
*2.3/ No need to interpret the protocol field relative to IPv4**
*S3.2.1:
The protocol number in interpreted relative
to the IP protocol that encapsulates the UDP packet (i.e. protocol of
the outer IP header).
IPv6 [RFC2460] defines the Next Header field to use the same protocol
identifier space as IPv4. There are no IPv4 protocol numbers that are
inappropriate for IPv6 (see the IANA protocol number registry
<http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml>).
Therefore, this should simply say that the protocol number is
interpreted as an IPv6 protocol number (and therefore the field would be
more appropriately called "Next Header").
*2.4/ No need to restrict interpretation of the protocol field**
*S3.2.1:
This draft should not state any restrictions (e.g. those in the second
and third paragraphs quoted below) that preclude certain protocol
numbers in combination with either an IPv4 or IPv6 outer.
For an IPv4 header the protocol may be set to any number except for
those that refer to IPv6 extension headers or ICMPv6 options (number
58). [...]
For an IPv6 header the protocol may be set to any defined protocol
number except Hop-by-hop options (number 0). [...]
Various implementations are capable of understanding an IPv6 extension
or v6-ICMP within an IPv4 header (e.g. [RFC6145
<https://tools.ietf.org/html/rfc6145#section-5.2>]). And any list of
restricted header combinations can never deal with newly defined
headers. So the only test needed is "Does your code for this combination
and order of headers have the logic for the next header?" GUE then only
needs to refer to the appropriate action already specified in RFC2046
(quoted below) rather than making up its own rules:
The Option Type identifiers are internally encoded such that their
highest-order two bits specify the action that must be taken if the
processing IPv6 node does not recognize the Option Type:
[...]
If, as a result of processing a header, a node is required to proceed
to the next header but the Next Header value in the current header is
unrecognized by the node, it should discard the packet and send an
ICMP Parameter Problem message to the source of the packet, with an
ICMP Code value of 1 ("unrecognized Next Header type encountered")
and the ICMP Pointer field containing the offset of the unrecognized
value within the original packet. The same action should be taken if
a node encounters a Next Header value of zero in any header other
than an IPv6 header.
There is a sentence at the end of S.3.6 (quoted below) that repeats
these unnecessary restrictions. If you agree with me, please also remove it.
[...] In this case next
header must refer to a valid IP protocol for IPv4. No other extension
headers or destination options are permitted with IPv4.
*2.5**/ Missed opportunity to liberalise interpretation of the protocol
field**
*
I believe that GUE offers the opportunity to liberalise, rather than
restrict, protocol field interpretation. In particular, GUE could allow
encapsulation of hop-by-hop options (next header number 0). You might
wonder what a HbH option could possibly mean within a GUE header - see
C2.4/ "GUE: a potential solution to the IPv6 extension header discard
problem" in my separate email about how to use GUE to solve the problem
where IPv6 packets with header extensions are highly prone to discard
[RFC7872 <https://tools.ietf.org/html/rfc7872>].
*2.6/ Positioning GUE with respect to existing IPv6 extension headers**
*
The draft needs to state rules for where GUE encapsulation fits in the
order of a chain of any IPv6 extension headers already present in an
arriving IPv6 packet. Below, this question is considered for both types
of encapsulation, and in both cases it can be seen that the UDP/GUE
header would not necessarily be the first header after an IPv6 outer.
* Network encap:
According to my reading of RFC2473, certain IPv6 extension headers in an
arriving IPv6 should (theoretically) be copied as extension headers for
the outer:
a) a Hop-by-Hop Options header (depending on the encap configuration,
but a jumbogram option would have to be copied)
b) a Routing header (depending on the encap configuration)
c) The Tunnel Encapsulation Limit Option (within a Destination
Options Extension Header)
- HbH options are pretty academic these days, given they cause about
39-54% discard [RFC7872 <https://tools.ietf.org/html/rfc7872>]. However,
if there is one on the inner, I guess we should still say that a GUE
network encap should copy it to the outer before UDP/GUE is added.
- I believe RFC2473 was wrong to say a routing header could be copied
to the outer. Imagine a packet gets tunnelled that has a routing header
listing addresses D2, D1 & D0 still left to visit. Although it is
unclear what it means to copy a routing header to the outer, it must
mean that these addresses would be visited by the tunnelled packet, then
visited again after decapsulation.
- I believe the Tunnel Encapsulation Limit Option is also pretty
academic these days, but again, if one arrived, a GUE network encap
ought to check the value, decrement it, and copy the header to the outer.
* Transport encap:
In this case, I have suggested where the UDP/GUE header should fit in
the following order of extension headers (copied from RFC2046):
IPv6 header
Hop-by-Hop Options header
+UDP
+GUE
Destination Options header (note 1)
Routing header
Fragment header
Authentication header (note 2)
Encapsulating Security Payload header (note 2)
Destination Options header (note 3)
upper-layer header
The draft ought to mention that if AH has been applied to a packet which
is then encapsulated by GUE in transport mode, the AH header is not
recalculated, so it does not cover the UDP/GUE headers. Decapsulation
works because the UDP/GUE headers are inserted before the authentication
header, so they will be removed (by a GUE decapsulator in transport
mode) before AH is verified.
Personally I don't know enough about routing headers to make the
decision on whether they should be above or below the GUE header in the
transport encap. I believe they are only processed when a packet reaches
the destination address in the main header, but I am not familiar with
all the different routing types (I know some are deprecated, and frankly
I couldn't be bothered to read the others).
*2.7/ Reliable delivery of control messages**
*
The examples of potential control messages (those with the 'C' flag)
given in S.3.5.1. (echo request/reply for testing) aim to mimic the data
channel, so unreliable delivery as a GUE datagram is appropriate.
The draft doesn't define any other tunnel control messages. However, if
it did, many/most would need to be delivered reliably and in order (e.g.
key agreement, any necessary configuration agreement, consistent
application of connection semantics, etc).
Therefore, reliable ordered delivery for control messages will need to
be defined (see C3.2/ "Reliable delivery of control messages" in
separate email for a suggested design).
*2.8/ Extensibility of the flags and optional fields scheme: doesn't work**
*S3.3:
This is meant to be "the primary mechanism of extensibility in GUE".
However, for extensibility to work, GUE needs to distinguish between:
* options: the base set of flags+options defined from the start and
required in all GUE code
* "extensions" (my term): future extensions to the flags and options.
The current GUE flags scheme only works for options, but it inherently
puts extensions into a chicken-and-egg stand-off. because:
a) S5.4 says an implementation MUST drop a packet with an unknown flag.
So, if the IETF later defines bit 7, until a very large proportion of
GUE decap implementations have been upgraded with logic that understands
bit 7, the packet is going to be dropped with high probability. So no
encap is going to want to set bit 7 on a packet, so there is no
motivation for a decap to implement the code for bit 7.
b) For such unknown flags, we cannot change "MUST drop" to "MUST
ignore", because the lengths of the fields are not self-describing -
they have to be hard-coded into an implementation. So if one GUE
implementation only has logic about the flags up to bit 6, but a packet
arrives with bit 8 set, the implementation doesn't know how large the
"Fields" field is, so it doesn't know where the private data starts.
For proper extensibility, each new GUE flagged option needs to be
self-describing, i.e. with additional fields to say:
a) Whether nodes that do not have the logic to understand the option
should drop or ignore the packet, separately for:
- nodes on the path
- nodes at the dest. (decap) of the GUE datagram.
b) Whether the option is intended to change on path (in which case it
should not be covered by integrity or authentication codes).
c) Whether the option should be copied or not by a GUE-in-GUE tunnel
encap (see A4.4/ "Tunnels in Tunnels" later).
d) The length of the option
e) Additionally you might want to borrow the IPv6 idea of controlling
whether there needs to be an error message or not, but personally I
believe that is overkill (the intention was for silent failure to be
impossible for critical features, but it is very hard to deliver error
messages reliably anyway).
The above shows that attempting to invent a new extensibility scheme
usually ends in tears. The IETF and others have developed
tried-and-tested extensibility approaches like TLV, CBOR. Even then,
they still have problems. The above points draw lessons from all this,
particularly:
* action codes and change codes in the initial bits of IPv6 HbH & DO
options [RFC2460]
* TRILL extension word flags: critical and non-critical separately for
hop-by-hop and ingress-to-egress (see [RFC7179] updated by [RFC7780]).
* 'Self-describing objects', including type and size, is listed as
'Architectural Principle of the Internet' number 3.12 in [RFC1958]
*2.9/ Hard-coded option lengths do not scale**
*
By hard-coding the length of each option in an RFC and in the GUE code
(rather than self-describing in the packet), you are stuck with a
certain size option for ever. Experience has proven that fields such as
message authentication codes (MACs), fragment IDs, etc. have to scale.
Admittedly, we could define flags for larger fields later, but I have
shown above that new flags would be undeployable.
*2.10/ Random access to options needs motivating*
Quoting S3.3:
Flags allow random access, for instance [...]
There might be a case for GUE to use a protocol heap rather than a stack
[Braden03]. If so, please motivate it.
[Braden03] Braden, R., Faber, T. & Handley, M., "From Protocol Stack to
Protocol Heap: Role-Based Architecture
<http://doi.acm.org/10.1145/774763.774765>," ACM SIGCOMM Computer
Communication Review 33:17--22 ACM (January 2003)
_*3/ STATE*__*
*_
*3.1/ Per-connection**state vs. ***stateless connections* but per-tunnel
state**
*
The GUE draft does not suggest a mechanism for GUE endpoints to apply
connection semantics.
* For transport encap the GUT draft suggests an approach that uses
per-flow state (see the example given in Figure 4 in
draft-manner-tsvwg-gut-02
<https://tools.ietf.org/html/draft-manner-tsvwg-gut-02#section-3.4>).
* For network encap a stateless approach is proposed in my separate
email (see C1/ "Stateless Connection Semantics"). Statelessness is
important to simplify migration during load-balancing, failures etc.
The 'shared fate' resilience principle [Clark88] maintains that a system
should avoid reliance on flow-state held on the path, preferring to hold
state solely at the endpoints. One could argue that, in transport encap
mode, the GUE endpoints are on the end hosts, and therefore, the
communication path is resilient because if GUE flow state is lost
because an end host fails, the communication will have failed anyway.
However, strictly, a GUE endpoint process is likely to be separate
(perhaps even in NIC hardware) so it could fail independently of the
true endpoint process of the connection.
So it would be ideal to use a stateless approach for both network and
transport encap. However, the best stateless approach I could come up
with (if it works at all) requires some coordination and hence one-off
set-up latency between the GUE endpoints. Therefore, stateless
connections will be:
* more appropriate for network encap (usually long-lived tunnels); and
* less useful for transport encap (opportunistic per connection).
To summarize, it is likely that the stateful approach will be used, at
least for some GUE encapsulators in transport mode. Therefore, for the
transport encap mode at least, the draft needs to consider per-flow
state and its management (see following section).
[Clark88] Clark, D.D., "The design philosophy of the DARPA internet
protocols," Proc. ACM SIGCOMM'88, Computer Communication Review
18(4):106--114 (August 1988)
*3.2/ Transport encap with **Connection Semantics: Flow state management**
*
Hosts already maintain flow-state for each connection in progress. To
support GUE in transport encap mode, it is trivial for the hosts at each
end to associate a little extra state with the existing state of each
inner flow:
* At the initiator end, it needs no flow-state to receive GUE packets,
but in order to send GUE packets, it associates the original (inner)
flow's ID with the source port it will use in the UDP outer to send
every GUE packet.
* At the responder end, it has to associate the inner flow ID with the
source port in arriving GUE UDP outer headers. It needs this so that,
when the inner flow sends out packets, the GUE encapsulator can
intercept them and encapsulate them with a GUE header, using the stored
source port as the destination port.
* Any error messages returned from the responder also need to be
encapsulated in the same way.
Also, the draft needs to specify:
* that a GUE transport decap ought to protect itself against DDoS by not
storing flow state if no associated socket is open;
* how long to time out unused flow state;
* what to do with a packet if the necessary flow state is not present;
*3.3/ Keepalives for middlebox flow state**
*
Middleboxes, such as firewalls and NATs time out the pin-hole associated
with UDP flow-state fairly rapidly, but rarely less than 15s [RFC5405].
RFC5405 rightly says that an application that uses UDP should be
responsible for recovering a timed out connection, rather than the stack
sending keepalives to hold open a connection, when it doesn't actually
know whether the application still wants the connection open.
Nonetheless, an inner flow will not be aware that it is being tunnelled
using UDP/GUE. Therefore it seems less inappropriate for the GUE encap
to keep state alive on behalf of the application, so it ought to send
keepalive GUE datagrams to hold any pin-hole open. However, if the
application has not sent anything for some time (whatever that means),
the GUE encap should time out the connection, rather than holding
middlebox flow-state (and its own flow-state) open for ever.
If you agree, it might be necessary to specify a keepalive control
message that a GUE encap can send to the remote end of the GUE tunnel
(which would also keep any flow-state at the remote end alive). These
would only be necessary in one direction, and would not need to be
reliably delivered.
See Section 3.1 of draft-manner-tsvwg-gut-02
<https://tools.ietf.org/html/draft-manner-tsvwg-gut-02#section-3.1> for
the keepalive control message defined for GUT.
_*4/ OPERATION*_
*
4.1/ Transport encap: to GUE or not to GUE?**
*
For transport encap, the draft needs to say how the host decides when to
use GUE and when not.
There's text on this inS.4 of the GUT draft
<https://tools.ietf.org/html/draft-manner-tsvwg-gut-02#section-4>, if
you want to use it.
*4.2/ Hop limit / TTL processing**
*I couldn't find any text about this. Perhaps you intended this sentence
in S.5.3 to cover it:
it should follow standard conventions for tunneling of
one IP protocol over another
I think it would be best to spell out Hop limit processing. There's text
on this inS.3.2 of the GUT draft
<https://tools.ietf.org/html/draft-manner-tsvwg-gut-02#section-3.2>, if
you want to use it.
*4.3/ Error messages**
*S5.4
No error message is returned
back to the encapsulator.
Please go through every type of error and in each case justify why no
error message to the encap is necessary.
*4.4/ T**unnels in tunnels
*S5.5 2nd para
It
may encapsulate a GUE packet in another GUE packet, for instance to
implement a network tunnel (i.e. by encapsulating an IP packet with a
GUE payload in another IP packet as a GUE payload).
A number of problems here:
1) A "GUE packet" has not been defined. I assume any UDP header with
either src or dst UDP port = 6080 (see A1.1/ "Inferring Connection
Semantics: the rule not the exception").
2) There is an incremental deployment problem here. Existing tunnels
won't check within the outer IP for whether a UDP port is a GUE port.
They will just add a new outer IP header without the UDP or GUE.
3) Whatever, if a tunnel is GUE-aware, this para needs to be clear
exactly which headers it should copy with the outer IP:
* Do you intend this to mean that all the following should be copied to
the outer IP header:
- the outer UDP,
- any v0 GUE header
- plus any GUE options or private data.
* Is it appropriate to copy all the options and private data? I think
only some (e.g. perhaps the VNID in certain circumstances?). Others
would not have the correct semantics if blindly copied (e.g. fragment
options, coverage of MACs, etc).
* How does a GUE-in-GUE encapsulator know which to copy?
Also, should any extension headers on an arriving IPv6 outer also be
copied to be associated with the new outer? If so, which ones, and how
does the encapsulator know? Do the same rules apply whether using
transport or network encapsulation?
I have been arguing since about 2009 that, when adding a new IP outer,
each IP (at least IPv6) extension header should self-describe which
headers should be copied to the outer on encap. At present RFC2473 lists
some extension headers that might be copied and says it depends on the
configuration of the encapsulator. But a hard-coded list precludes
introduction of any new extension that needs to be copied. And certainly
it doesn't work for extensions like GUE that don't fit into the original
mould of what an IPv6 extension looks like. The behaviour needs to be
somehow self-declared in each header, not in a standard.
It is tough to solve this problem in a way that will work with existing
tunnels. It needs solving more generally, not just for GUE. However, as
long as GUE encapsulators address this problem from day-1, GUE presents
an opportunity to solve the general problem in environments where all
encapsulations are GUE-based (see my proposed solution in C4.1/
"Ensuring certain GUE headers are copied when a GUE packet is tunnelled"
within my separate email on redesign). Then other encapsulation
approaches might follow.
*4.5/ **SHOULD adjust MTU?
*
An operator may set MTU to account for encapsulation overhead
and reduce the likelihood of fragmentation.
I would expect "SHOULD" here.
You might want to refer to draft-ietf-tram-stun-pmtud for a way to do
PMTUD with UDP (for STUN, but I think it would be similar for GUE).
*
**4.6/ Is orig-proto field necessary in the fragmentation option?*
S4.3 of draft-herbert-gue-extensions-00
Why does the original protocol of a fragmented packet need to be visible
before reassembly by declaring it in the GUE fragmentation option of
each fragment? The GUE protocol field will be available once the
fragments are reassembled, and I can't see why it would be needed before
that.
It is not good security practice to create multiple fields that are all
intended to be set to the same value. Even if the implementation uses
these orig-proto fields before reassembling the fragments, it will still
have to check that they all match the GUE protocol field when the packet
has been reassembled. And if any are not the same, it will raise
security concerns about any action that had previously been taken based
on an inconsistent value.
*4.7/ Congestion Control: reductio ad absurdum**
*S5.9
I suggest you remove the para about DCCP being appropriate for tunnel
congestion control. I appreciate you are trying to comply with RFC5405,
but it is impossible for tunnel specs to do so without looking absurd.
The more you try, the more it will look like you are the ones that are
absurd. RFC5405 gives no guidance on how to comply with its requirement
about congestion control of non-IP traffic across a tunnel... because
there is no running code for tunnel congestion control, or for a network
circuit breaker.
It has been suggested in the past that DCCP should be used across
tunnels. DCCP is intended for a single flow and all the DCCP profiles
defined so far ensure a DCCP "flow" will consume about as much capacity
as a TCP flow. If DCCP were to be applied across a GUE tunnel it would
reduce the rate of the aggregate of all flows across the tunnel to
roughly the same as a /single/ TCP flow (see the intro of RFC7893
"Pseudowire Congestion Considerations").
One might imagine that RFC5405 means that a tunnel protocol designer
would have to detect roughly how many flows a tunnel aggregate consisted
of at any one time (say N flows) and attempt to design a congestion
control (e.g. a DCCP profile) to consume roughly as much capacity as N
TCP flows. However, this would probably cause horror for some in the
transport area at the thought of the IETF endorsing a congestion control
that can be N times as greedy as TCP.
To further reduce the idea of a tunnel encap applying congestion control
to absurdity, it would need:
a) a huge buffer to absorb incoming packets whenever they arrived faster
than the tunnel rate. All packets (in small and large flows) would back
up behind this huge queue, which would be called buffer bloat, which
would cause horror for most people in the transport area.
b) ideally, a time machine (a negative buffer) to bring packets forward
in time whenever the arrival rate of all the flows was insufficient to
satisfy the desired aggregate rate of the tunnel.
c) the addition of feedback channel(s) and a huge amount of extra
processing.
[As you can see, I don't support the idea in RFC5405 that a tunnel
becomes responsible for congestion control of traffic that it
encapsulates. Otherwise, to be consistent, an Ethernet link would become
responsible for congestion control of traffic it encapsulates. However,
I accept that consistency with RFC5405 is currently a hurdle your draft
has to cross before it can be approved. If you feel you have to suggest
a mechanism, IMO a policer makes sense - either a rate policer or a
congestion-rate policer.]
*4.8/ Multicast outer -> Implosion on inner destination**
*S.5.10
Consider an inner flow of unicast packets, src-IP A, dst-IP B. Consider
the encap adds an outer addressed to multicast address M, and consider n
decapsulators subscribe to group M. This will cause the network to
duplicate each packet n times. As each decap forwards the inner, n
duplicates of each packet will converge on B.
This might make sense with unicast inner packets for a small number of
decaps (e.g. two for redundancy). And a multicast overlay could make
sense for multicast inner packets as long as the multicast routing was
aware of the P2MP tunnel (with suitable grouping of multicast groups).
I think the text should say that a multicast outer is not precluded,
because it is a theoretical possibility, but it should not be attempted
without a safety harness and an empty bladder.
*4.9/ Deriving flow entropy from the inner is contrary to "GUE permits
encap of arbitrary IP protocols" claim**
*S.5.11.1
The general idea for creating flow entropy seems to be for the GUE encap
to map inner flows of possibly "atypical IP protocols" to individual UDP
outer flows, on the assumption that switches or routers that implement
ECMP etc. will understand UDP but not "atypical IP protocols". Let's
examine this claim by taking network encap and transport encap separately.
1) Network encap
Imagine that a GUE encap has been implemented that understands TCP, UDP,
SCTP, DCCP, ICMP, RSVP, IPsec and ESP.
Then researchers implement NewSexyTP, with a new IP protocol number.
Every GUE encap in the world doesn't have any logic to understand or
locate the flow ID fields of NewSexyTP. So GUE does not "permit encap of
arbitrary IP protocols" as claimed in the motivation section.
Further, why will GUE implementations be updated with logic to
understand NewSexyTP any faster than the ECMP code in general-purpose
switches and routers? One GUE implementation might be updated, but other
developers might not so diligently track the latest transport protocols.
One cannot even really argue that the ECMP code in switches and routers
is implemented in hardware, so it will be harder to change than GUE
code. Because the forwarding performance of GUE tunnel encap will need
to be no different to the performance of forwarding in general switches
and routers, so if hardware is necessary for one it will be necessary
for the other.
2) Transport encap.
If GUE encap is implemented as a centralized daemon process on a host or
centralized in a NIC, it will suffer from the same lack of forward
compatibility with new transport protocols as the network encap -
particularly if it is implemented in NIC hardware. Ie, if an operator
installs SexyNewTP in their OS, they will also have to wait for a GUE
update that supports SexyNewTP. This is the case with or without
connection semantics.
However, it might be possible to implement GUE transport encap
(including with connection semantics) so that each instance of a
protocol stack is associated with an instance of GUE (warning: I have no
idea yet whether this will be possible). In this case, each GUE instance
would consistently add the same outer port number to the inner protocol
instance it was associated with, without needing to understand how to
identify a flow ID in any particular protocol.
In summary, certainly for net encap, but possibly not for transport
encap, GUE only helps "atypical IP protocols" that a particular GUE
encap implementation already understands.
*4.10/ Flow entropy from encrypted data could weaken the crypto?**
*S.5.11.1
o If a node is encrypting a packet using ESP tunnel mode and GUE
encapsulation, the flow entropy could be based on the contents
of clear-text packet. For instance, a canonical five-tuple hash
for a TCP/IP packet could be used.
I'm not a crypto expert, but it sounds dangerous to take some clear-text
from a known position in the data, hash it with a function that is not
strongly one-way, then send this hash along with the cipher text.
I think the SPI can be used as a unique consistent per-flow value, can't
it? The SPI has been suitably randomised so that it reveals nothing
about the flow ID.
*4.11/ No need to constrain flow entropy distribution**
*S.5.11.2
o The flow entropy should have a uniform distribution across
encapsulated flows.
Equal distribution of flows is not necessarily appropriate for all
scenarios. Flows have a distribution of sizes, and altho ECMP is
generally done randomly, an operator might want to (somehow) bias the
hash algorithm to allow for the flows with the highest rate, which might
otherwise unbalance the load. See for instance:
"Engineered Elephant Flows for Boosting Application Performance in
Large-Scale CLOS Networks
<https://www.broadcom.com/collateral/wp/OF-DPA-WP102-R.pdf>" Broadcom
White Paper (March 2014)
*4.12/ No need to constrain flow entropy interpretation**
*
Decapsulators, or any networking devices, should not attempt to
interpret flow entropy as anything more than an opaque value.
This seems unnecessarily constraining. This might not be a good idea,
but if someone finds a use for it, there's no need to stop them - if
it's useful they'll ignore you anyway, so why bother saying it? Perhaps
you intended to explain why doing this could be problematic, rather than
precluding it?
_*5/ SECURITY*__*
*_
*5.1/ Addresses that are both visible and hidden? Have your GUE and eat
it too?**
*
S.7. In the following sentence,
Existing network security
mechanisms, such as address spoofing detection, DDOS mitigation, and
transparent encrypted tunnels can be applied to GUE packets.
This should point out that an existing set of address spoofing detection
rules would not work with GUE. I think you meant that existing rules and
mechanisms could be modified to check the packets encapsulated by GUE
without using radically new techniques.
However, if GUE is in network encap mode and it encrypts the IP headers
of the inner packets, address spoofing detection and DDoS mitigation
will not be possible over the length of the GUE tunnel. You cannot both
claim that GUE can hide information, and that GUE allows existing
security techniques to work that rely on access to the hidden information.
*5.2/ How can the Security option protect a UDP/GUE header from being
moved or removed?**
*
The Security option is "used to provide integrity and authentication of
the GUE header."
I assume you envisage this would be complemented by other authentication
techniques such as IPsec AH to provide integrity and authentication of
the rest of the packet.
However, it occurs to me that the two together do not protect the
integrity of the /structure/ of the packet as a whole (whether network
or transport encap). An on-path attacker could still move the UDP/GUE
header within the packet (it might be possible to construct a valid
packet with altered semantics), or remove the UDP/GUE header completely.
I can't immediately think whether any damage could be done with such an
attack, or how to prevent it. However, I'm sure there will be a crypto
expert for whom this is not a new problem.
Also, the 32B max length of the security option is insufficient. I
looked for a MAC protocol where a larger field is needed, and the first
one I picked required a larger field: RFC4383 "TESLA in Secure RTP"
requires 34B, and that's just for the default sizes, not even the
maximum. I picked TESLA because I knew each datagram needs a lot of
authentication space. TESLA provides multicast message authentication,
so as well as a key index and a MAC, each packet reveals a continually
changing key.
*5.3/ What happens when a port scan sends a datagram to port 6080?**
*
When a port scan (that doesn't necessarily know about GUE) sends a
datagram to port 6080, if the datagram has a body, and the body starts
with a zero bit, the GUE daemon will start processing it.
If the first 4 octets happen (randomly) to be set to values that would
be a valid GUE header (see S.5.4), it will be decapsulated and forwarded
to a protocol handler.
Not a show-stopper, but worth documenting?
*5.4/ Firewalls will still block new/atypical protocols**
*Few firewalls allow incoming UDP. So GUE will not enable deployment of
servers using atypical/new protocols, which will still face a deployment
problem.
If a firewall opens a pin-hole to allow incoming UDP to access the
well-known GUE port it would allow attackers to reach servers of any
protocol while bypassing the firewall. E.g. an attacker could access a
TCP-server by encapsulating TCP in GUE in order to bypass the firewall.
Therefore, a firewall will only open a pin-hole to a GUE server, if it
also inspects the packet encapsulated by GUE and applies all its normal
rules to that as well.
This is why I have said elsewhere that the draft should state that
firewall bypass by new/atypical protocols is a non-goal of GUE.
*5.5/ Transport Encap: Two Passes through a Local Firewall?**
*
GUE in transport mode resubmits the encapsulated packet to the host's IP
stack. But it needs to make sure it re-injects the packet at the correct
point in relation to any local firewall.
* If the firewall includes rules to inspect the packet encapsulated with
GUE (as discussed in the previous point), it would make sense to
re-submit the packet above the local firewall.
* If not, GUE should resubmit the packet so that it passes through the
local firewall again.
The latter mode would make more sense if GUE was also decrypting the
inner packet. So, rather than have two options, a local firewall could
work co-operatively with GUE in transport mode, so it doesn't have to
inspect the inner in both passes.
*6/ Implementation**
*
*6.1/ Practical Large Receive Offload Requirements**
*Appendix A.4 says:
The conservative approach to supporting LRO for GUE would be to
assign packets to the same flow only if they have identical five-
tuple and were encapsulated the same way. That is the outer IP
addresses, the outer UDP ports, GUE protocol, GUE flags and fields,
and inner five tuple are all identical.
Rant: It is sad if such a conservative approach to LRO is still
necessary. Any API to LRO hardware needs to be able to be given the
locations of certain header fields that are deliberately intended to
vary, so it can offer the facility to separately report these for each
packet. A MAC of the encapsulating headers is a good case in point. ECN
is an even better example of a varying field, because it has been a
standard part of the IP header since 2001, long before LRO hardware was
designed.
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
_______________________________________________
nvo3 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/nvo3
