Mike, We had a Telecom Cloud Side Meeting during IETF120. The proponents of the side meeting want to start some work on the operation aspect of placing workload in Telecom Clouds. Some of the issues you have raised might be useful as the use cases for their proposed work. Please see this link of the presentations: https://ietf.webex.com/ietf/ldr.php?RCID=1e23d95fd88b71da5748443786a0e414
Linda From: Mike Ounsworth <mike.ounswo...@entrust.com> Sent: Wednesday, September 11, 2024 7:40 PM To: Joel Halpern <jmh.dir...@joelhalpern.com>; sec...@ietf.org; Deb Cooley <debcool...@gmail.com>; Tero Kivinen <kivi...@iki.fi> Cc: draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org; rtgwg@ietf.org Subject: RE: [EXTERNAL] Re: Seder early review of draft-ietf-rtgwg-net2cloud-problem-statement-41 You may own the workload in the cloud (VM, container, lambda function), but you don’t own the infrastructure that it’s running on – that’s the whole point of a public cloud; it’s your job running on someone else’s hardware. In fact, it’s your and several million other people’s jobs running on someone else’s hardware. See, for example, the Meltdown and Spectre attack for an example of malicious actors jumping across workloads running on the same CPU [1]. If you shipped your routing equipment to <insert government-funded hacker group here> for a month and then got it back, would you put it back into your datacenter? The publicly-accessible infrastructure within a public cloud is probably not far removed from that in terms of threat model. I would bet money that various access-as-a-service groups have undetected rootkits on all sorts of public cloud infra. The extent to which you want to trust that, and the extent to which you want to expose your network to that, is a risk decision to be made by the network operators who want to do the bridging, and risk tolerances will vary from “I’ve been told to expose these worker nodes, and I don’t want to expose anything more”, to “This cloud company runs a tighter ship than us, no concerns”, with probably several reasonable discrete steps in between. I am not a domain expert, so I can’t recommend any specific text changes, but I’m sure that at least some discussion of this risk slider belongs in the draft, and would apply to basically every section of this draft. [1]: https://meltdownattack.com/ --- Mike Ounsworth From: Joel Halpern <jmh.dir...@joelhalpern.com<mailto:jmh.dir...@joelhalpern.com>> Sent: Wednesday, September 11, 2024 8:43 PM To: Mike Ounsworth <mike.ounswo...@entrust.com<mailto:mike.ounswo...@entrust.com>>; sec...@ietf.org<mailto:sec...@ietf.org>; Deb Cooley <debcool...@gmail.com<mailto:debcool...@gmail.com>>; Tero Kivinen <kivi...@iki.fi<mailto:kivi...@iki.fi>> Cc: draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org<mailto:draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org>; rtgwg@ietf.org<mailto:rtgwg@ietf.org> Subject: Re: [EXTERNAL] Re: Seder early review of draft-ietf-rtgwg-net2cloud-problem-statement-41 This may help, if I am reading this and the draft correctly, point to a place to clarify. The assumption is that the communicating parties, while one is hosted locally and one is hosted in the cloud, are actually all acting on behalf of the This may help, if I am reading this and the draft correctly, point to a place to clarify. The assumption is that the communicating parties, while one is hosted locally and one is hosted in the cloud, are actually all acting on behalf of the enterprise. The enterprise is trying to interconnect its workloads. And the connectivity is in support of that. Not random traffic to or from the enterprise. It is a little more complicated because the connectivity is often provided by several third parties (ISPs, Data Center operators, ...) Tha authors would need to confirm that my reading is correct, but if it is, would it help your concerns if more emphasis on the communicating parties, with some indication of how the cloud end restricts those communicating entities, were added to the document? Yours, Joel On 9/11/2024 8:59 PM, Mike Ounsworth wrote: Hi Joel, Full disclosure: I’m a crypto and web application guy. I have some exposure to DNS and TCP, but I had to google “BGP” while reading this draft. I tried my best, but I probably don’t have even the minimum required background knowledge to be security reviewing this draft. My (limited) understanding of DNS Resolvers is that if you bridge networks such that the foreign network has access to your network’s DNS resolvers, then agents on the foreign network can now query any FQDN and find out if it exists on your network and maybe deduce some topological information based on IP address (aka, “map your network”). I could well imagine that in the type of net2cloud setups you are describing, you would want the cloud to be able to resolve certain workloads on the on-prem network (ex.: build agent worker nodes), but not necessarily everything on your network (ex.: laptops and workstations). I tried to provide the example that just knowing that, for example, all the workstations under the domain “corp.acme.com” suddenly become resolvable on your corporate network might indicate a merger & acquisition that is not public knowledge yet. To my understanding, DNS is not really designed for privacy, ie to hide the existence of certain subnets depending on who the requester is (unless you do some super complicated recursive DNS setups). Maybe Linda is right and this is out-of-scope for this document, but it feels to me like some discussion of this point would fit into section 3.5. Similarly, my understanding of NATs is that they function by having a gateway proxy traffic from its subnet to a wider area network (or vice versa from a wide area network into a subnet), such that it appears to firewalls and other traffic monitoring software as if that traffic originates from the gateway itself. Being overly cavalier about using NATs to bridge trusted and untrusted networks feels like it would really hamper security traffic monitoring tools and firewalls from being able to properly enforce origin / destination based rules. Maybe Linda is right and this is out-of-scope for this document, but it feels to me like some discussion of this point would fit into section 3.6. I am far less familiar with the other technologies mentioned in the rest of sections 3, 4, and 5, but I would assume that anywhere that you’re bridging a trusted network (aka a network you own) to an untrusted network (aka a network you don’t own), you want to be a bit careful about what suddenly now has access to what. In my $dayjob, we often perform security modelling exercises where we look at a network component, color it red and say “Ok, that’s compromised, what can it see? Now color all of those red”. Granted, as Linda said, we are largely thinking about application layer here, but still, vulnerabilities and exploitable configurations can and do exist at all layers and in this regard, subnet boundaries are extremely important for containing the spread of bad things and relaxing subnet boundaries should be done with extreme caution. In that this is a problem statement document, I’m sure there is something you can say in each section to this point. --- Mike Ounsworth From: Joel Halpern <j...@joelhalpern.com><mailto:j...@joelhalpern.com> Sent: Wednesday, September 11, 2024 7:14 PM To: Mike Ounsworth <mike.ounswo...@entrust.com><mailto:mike.ounswo...@entrust.com>; sec...@ietf.org<mailto:sec...@ietf.org>; Deb Cooley <debcool...@gmail.com><mailto:debcool...@gmail.com>; Tero Kivinen <kivi...@iki.fi><mailto:kivi...@iki.fi> Cc: draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org<mailto:draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org>; rtgwg@ietf.org<mailto:rtgwg@ietf.org> Subject: [EXTERNAL] Re: Seder early review of draft-ietf-rtgwg-net2cloud-problem-statement-41 I know and respect that you asked that a different reviewer be appointed. However, as Shepherd I am trying to figure out if there is an underlying problem that Linda and I were not able to get at from reading your review. The document is about I know and respect that you asked that a different reviewer be appointed. However, as Shepherd I am trying to figure out if there is an underlying problem that Linda and I were not able to get at from reading your review. The document is about the problems edge devices encounter inclassifying traffic that they are handling to and from data centers (thus net2cloud). Apparently, from the comments I think I understood, you are expecting fine grained application level differentiation for security purposes to be part of the problems this draft addresses. 1) Did I misunderstand what you expected / wanted / sought? 2) Can you help us understand what wording in the document led to that expectation, so we can clarify the document? Thank you, Joel Halpern (document Shepherd) On 9/11/2024 11:56 AM, Mike Ounsworth wrote: Hi Linda, Alright, you’ve rejected every one of my review comments as being out-of-scope. I clearly lack the domain knowledge to review this document properly. @Deb Cooley<mailto:debcool...@gmail.com> or @Tero Kivinen<mailto:kivi...@iki.fi> can you please re-assign this SecDir review to a different reviewer who has the appropriate Routing domain expertise? --- Mike Ounsworth From: Linda Dunbar <linda.dun...@futurewei.com><mailto:linda.dun...@futurewei.com> Sent: Tuesday, September 10, 2024 2:53 PM To: Mike Ounsworth <mike.ounswo...@entrust.com><mailto:mike.ounswo...@entrust.com>; sec...@ietf.org<mailto:sec...@ietf.org> Cc: draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org<mailto:draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org>; rtgwg@ietf.org<mailto:rtgwg@ietf.org> Subject: [EXTERNAL] RE: Seder early review of draft-ietf-rtgwg-net2cloud-problem-statement-41 Mike, Thank you very much for the review and the comments. Please see below the resolutions to your comments and suggestions. Linda -----Original Message----- From: Mike Ounsworth via Datatracker <noreply@ ietf. org> Sent: Tuesday, September Mike, Thank you very much for the review and the comments. Please see below the resolutions to your comments and suggestions. Linda -----Original Message----- From: Mike Ounsworth via Datatracker <nore...@ietf.org<mailto:nore...@ietf.org>> Sent: Tuesday, September 10, 2024 7:11 AM To: sec...@ietf.org<mailto:sec...@ietf.org> Cc: draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org<mailto:draft-ietf-rtgwg-net2cloud-problem-statement....@ietf.org>; rtgwg@ietf.org<mailto:rtgwg@ietf.org> Subject: Secdir early review of draft-ietf-rtgwg-net2cloud-problem-statement-41 Reviewer: Mike Ounsworth Review result: Has Issues I have reviewed this document as part of the security directorate's ongoing effort to review all IETF documents being processed by the IESG. These comments were written primarily for the benefit of the security area directors. Document editors and WG chairs should treat these comments just like any other last call comments. The summary of the review is that while the Security Considerations section lists a few specific good points, it does not address the more fundamental issue that when you bridge a network you own to a network that you don't, you should consider at ever level how much and how deep you're intending that access into your network to be. -- For example: I may want O365 to see my on-prem Exchange server, but not to be able to build a network map of all corporate laptops and workstations. [Linda] O365 is application level, which is out of the scope of this document. This document identifies high-level problems that can be addressed by protocol extensions within the IETF routing area domain. Other Cloud DC problems (e.g., managing Cloud spending) are out of the scope of this document. This review is broken up into "Security Comments" first, and then "Editorial Comments". Security Comments For some levity, I will start my security review with a comic that I think brilliantly summarizes the situation https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxkcd.com%2F2044%2F&data=05%7C02%7Clinda.dunbar%40futurewei.com%7C91333bfec45a4c29563b08dcd1a25a85%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C638615742483976543%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=2a3pwixOSdvzVMTh62K%2Bx%2F5vp8%2B8lIIat2ObG5E6Y5E%3D&reserved=0<https://urldefense.com/v3/__https:/nam11.safelinks.protection.outlook.com/?url=https*3A*2F*2Fxkcd.com*2F2044*2F&data=05*7C02*7Clinda.dunbar*40futurewei.com*7C91333bfec45a4c29563b08dcd1a25a85*7C0fee8ff2a3b240189c753a1d5591fedc*7C1*7C0*7C638615742483976543*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C0*7C*7C*7C&sdata=2a3pwixOSdvzVMTh62K*2Bx*2F5vp8*2B8lIIat2ObG5E6Y5E*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!FJ-Y8qCqXTj2!e-q0IrunoecSiqEVNkzb0uuZ3oecLCoBiDnC84-iaEDIDKKAK8i8WjOs_jQARHzhWRy5OeKz-URJo6fbgD5ILHQ-hkDSiA$> This draft is hitting the "I wish these parts could communicate more easily", "Integrate Everything!" boxes, and it's addressing some of the "Oh-oh, there are so many connections. It's creating Bugs and Security Holes" box, but it's not addressing the final box that often (especially for security) segmentation and sandboxing is a good thing, and you need to be mindful of keeping that segmentation where it matters. [Linda] Thank you very much for sharing your brilliant comic. It is very interesting. But those sandboxes are out of the scope of IETF Routing Area. Secton 3 This document would benefit from some discussion of thinking about where your "sensitive" applications and data is, and what needs to be protected from what. [Linda] That would need another document in IETF Application Area to describe “sensitive” applications and data placement in Cloud. The Net2Cloud document is to describe high-level problems that the IETF Routing area can address. Absent is discussion that sometimes the drive to multi-cloud or hybrid-cloud is actually motivated by security. Ex1: sometimes you have to run super risky workloads (like user-submitted code or virus samples), and you know that cloud providers can build a better sandbox than you can, so the motivation is to get the risky stuff off your on-prem network. Ex2: You may put a highly sensitive server in the cloud for its own protection so that it is isolated from a potential compromise of your on-prem network. Ex3: sometimes you want to keep your sensitive data (ex.: financial, contracts, pre-patent products, etc) on-prem so that you can have extremely tight control of it. Depending on whether you're trying to protect on-prem from cloud or protect cloud from on-prem, this leads to different design considerations across pretty much every section of this document. [Linda] All those problems you have identified are valid and need guidelines. But they are not within the IETF Routing Area. Probably consider writing a document in IETF Ops Area to address issues and solutions for workloads in hybrid Clouds. Section 3.1: TL;DR: I am not an expert on GBP, but it very much feels like there should be at least some relevant security considerations in this section. [Linda] The security considerations for network, like using IPsec/MPLS, to connect to Cloud GW are in Section 7. I am far from an expert in BGP, but my quick google shows that BGP Peering is traditionally symmetric; ie the goal is to fully bridge the two networks. In an On-Prem -> Cloud setup, you probably want to consider separately whether you want to grant on-prem things access to the cloud network vs granting cloud things access to the on-prem network. You also want to consider whether you're intending to grant access to *the entire* on-prem network, or only certain subnets of it. This becomes a security issue pretty quickly if you unintentionally grant workloads in a public cloud with broad access to your on-prem network. This falls inline with the sentence that is already in the draft: "As such, there is pressure to peer more widely with more customers, including those who may lack the expertise and experience in running complex BGP peering relationships. This can contribute to ..." because doing partial bridging / network exposure is certainly more complex. It also begs the question of whether the public cloud operators are properly isolating BGP-peered customers from each other. [Linda] Yes, isolating BGP-peered customers can be achieved by VPN, BGP Communities for Traffic Engineering and Filtering. For example, Azure allows the use of BGP communities to define the scope of route advertisements in their peering configuration. Similarly, AWS Direct Connect lets you tag routes with BGP communities to control their propagation. The draft-ietf-rtgwg-net2cloud-problem-statement- is NOT intended to describe all the existing procedures for BGP peering. But to identify some issues that can be solved by protocol extensions within the IETF routing area domain. Secton 3.5 Similar to my comment for 3.1: the described scenario is about using your on-prem DNS servers to bridge two cloud DNS domains, and sorta implied is that cloud-based workloads may need to resolve on-prem resources. Absent is a discussion that you may not want cloud-based workloads to be able to resolve (from a security perspective, "map") your entire on-prem network; for example, even if sensitive on-prem resources are not reachable due to TCP/UDP firewalling, the ability for an attacker to build a network map of hostnames and addresses may already be compromising in some situations for example addition of certain hostnames / domains to a local network may indicate a not-yet-publicly-announced merger, acquisition, department, partnership, or new product feature. That makes DNS Practices for Hybrid Workloads even more complex if you only want to expose *part* of your on-prem DNS space to the cloud workloads. [Linda] the problem you described here are application layer problem. Not within the scope of this document. There should be some discussion of confidentiality: whether it's an on-prem component resolving an cloud component, or vice-versa, these are private internal components, and the DNS queries should be considered sensitive metadata. So cross-network DNS queries should be paired with an encryption layer such as DNS-over-HTTPS, or ensuring that DNS queries are routed over a site-to-cloud VPN. Section 3.6 Surely we could think up at least half a dozen security considerations relating to overly-broad NATs punching inintentional holes in firewalls. This is the section where the XKCD comic really applies. [Linda] interesting problem. But out of the scope of this document. Are you sure you want your cloud workloads to have outbound to the internet? That enables compromised workloads to "phone home" to the attacker's command-and-control server or reverse-shell server, or to exfiltrate data to the attackers server, or .., or .., or. Typically on-prem enterprises have Data Loss Prevention (DLP) tools to detect this sort of behaviour leaving their network, but public clouds may or may not have enterprise-grade DLP solutions that monitor outbound connections from workloads for suspicious traffic patterns. We could also ask about whether compromised cloud workloads should have access to resources outside their local subnet. In security modelling, we typically consider subnets of workloads to be nice security containers ... at least a compromise can't spread beyond a subnet boundary; but if you start using NAT to punch holes in your firewalls, then you don't have firewalls anymore and all bets are off. We could also ask about whether it's a good thing for things outside a VPC to be able to reach things inside. Typically a multi-tiered cloud application will bury sensitive things like databases, HSMs, secrets vaults, legacy applications that don't meet modern security requirements, etc deep in a backend VPC for their own protection. Again, if you start using NAT to punch holes in your firewalls, then you don't have firewalls anymore. ... speaking of the word "firewall", it only appears once in the entire document (in the Introduction). I would think that this word would feature prominently throughout pretty much every section of a document that is fundamentally about bridging networks of things you own and things you don't. [Linda] very interesting problem. Which IETF area do you think we can propose solutions? Secton 3.7 This also has security concerns in that inbound tunnels to on-prem networks often want to have IP based firewalling, or increasingly AI-based anomaly detection, but constantly-shifting cloud sides of the tunnel make this sort of thing hard. [Linda] interesting problem. But out of the scope of this document. Sections 4 & 5 Nothing new that isn't already mentioned above. Editorial Comments (I am a complete outsider to the routing space, so feel free to disregard editorial comments if these points would be obvious to the target audience of this document). 1. Introduction Is it correct to interpret from the last sentence that the target audience of this document is network admins for existing enterprise networks who are considering adopting some form of hybrid cloud? If so, this point becomes muddier further on as some of the proposed mitigations seems like they need to be implemented by the the public cloud infrastructure. Possibly the document could benefit from more clearly separating "on-prem components", "cloud components under the control of the customer" and "cloud infrastructure under the control of the cloud operator", and being clearer about where each suggested mitigation applies? As a concrete example of this, Section 3.3 has the sentence: " One method to mitigate the problems listed above is to use anycast [RFC4786] for the services so that network proximity and conditions can be automatically considered in optimal path selection." which, at least to my layman's reading, is not clear whether this is something the customer can do, or needs to be done by the cloud operator. And then the other two suggested mitigations in that section really sound like things the cloud operator need to implement. [Linda] The intent of this document is to identify issues that can be addressed by routing protocol extensions. So, the audience should be IETF Routing Area participants. Section 2: Is it possible to expand the acronym for "SD-WAN"? It's used several times throughout the document but never defined. I assume it's "Software-Defined Wide Area Network"? [Linda] You are correct. SD-WAN is defined in Section 2. The term "BGP peering" is first used in 3.1, but I am unfamiliar with it, and it was not defined in 2, it probably should be. [Linda] BGP is the cornerstone of routing and a fundamental protocol within the IETF community. BGP peering is well known. Section 3.2 Term "IGP" used without any explanation or reference. [Linda] IGP is another base routing protocol that is well known. Term "EVPN" used without any explanation or reference. [Linda] as stated in the document, RFC7432 describes the details of EVPN. Section 3.3 This section uses the terms "DNS Server", "client", "DNS resolver", and "Local DNS resolver". It might enhance readabilty if the text was explicit about which things are part of the on-prem network (and therefor can be mitigated by on-prem solutions), and which are part of the cloud. [Linda] All those terms can be either an on-prem network or the cloud. Secton 3.6 I had to read this sentence 3 times to parse it: > By configuration, some private subnets can have NAT functionality to reach out to external networks, and some private subnets are internal to a Cloud DC only. I think a simple word change would help: > By configuration, some private subnets can have NAT functionality to reach out to external networks, *WHILE* some private subnets are internal to a Cloud DC only. [Linda] Thanks for the suggestion. Changed in -v42 per your suggestion. Section 5.1 > in [Section 3] (Security Considerations) The link "[Section 3]" does not appear to go to the Security Considerations? [Linda] Since this document is in the Routing Area, only one section describe the Security Consideration for the potential routing protocol extensions. Linda
_______________________________________________ rtgwg mailing list -- rtgwg@ietf.org To unsubscribe send an email to rtgwg-le...@ietf.org
