I have discovered the solution to my problem: - I still need a NAT type guest network (rather than an "open" or "route" type). - I discovered the *iptables rules* that was causing me all this trouble, and deleted them and everything works as desired.
For a "more permanent solution", I have used a libvirt network hook ( https://www.libvirt.org/hooks.html ) to remove the iptables rule on certain libvirt network events. If someone else in the future comes along with a similar problem, see here for the hook script: https://gist.github.com/cbluth/61753e8b2568d08294efe556e55246bf#file-etc_libvirt_hooks_network I have a general question for the mailing list regarding the libvirt network hook: As this hook is called on certain events, it deletes the iptables rule. But, how did the rules get there in the first place? Would there be a race condition between the event of deleting the iptables rule and the event of libvirtd creating the iptables rule? Is there a possibility of rejected packets because of this potential race condition? I have searched for some info regarding trying to prevent the rules from being applied in the first place, but I havent found much (or perhaps I am using the wrong search terms). Anyways, my original issue is fixed, but any info regarding my questions above is appreciated. Thanks, -Cobin On Thu, May 31, 2018 at 3:59 PM Cobin Bluth <cbl...@gmail.com> wrote: > Hi Peter and other Libvirt-Users, > > Thank you, I greatly appreciate your response. The diagram I provided may > be a little misleading, so I have attached a new diagram (also available > here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide > some more detail about my scenario here: > > These physical nodes are provided to me by "bare-metal cloud" providers > like Scaleway / OVH / Packet.net / etc. > I have opted to use Ubuntu on each bare-metal server, but I am also > well-versed in RHEL/CentOS environments, so if you want to help me out by > telling me CentOS-specific details, I'm sure that I can "translate" that to > an Ubuntu/Debian setup. Considering my scenario with bare-metal providers, > I do *not* have the flexibility to add/remove physical NICs as I need > them, I am sort of "locked in" to a single-IP/single-NIC configuration for > each physical node. > > But, I **do** have the flexibility of adding more physical nodes as I > need them, and I would like to leverage this option when I want to expand > my cluster. Its easy for me to buy a bare-metal cloud server (takes a few > minutes), and setting up a 1U/2U server by myself on a "lab" network could > take me a few hours or so (not to mention, I dont have the resources to > host my own network for this). Also, I need to account for the fact that > hardware *will* fail somehow eventually, so adding/removing nodes would > need to be "easy" in the way that I shouldnt have to configure new DHCP > scopes on all the other nodes when I decide to add one more node to the > cluster. Also, as a side note, I do not intend to have any central or > shared volume pool between the nodes, all storage will be direct on a local > LVM volume, and the ability to perform live migrations of guests between > hosts is not a high priority for me, I intend for each guest to be > *"stationary" > to each host*. > > Each node's *eth0* is connected via IPSec (unicast and transport mode) to > the other. This is to encrypt the communication between all nodes because > it travels over the internet. Additionally, each node currently has a > *vxlan0* interface in the *172.20.0.0/24 <http://172.20.0.0/24>* network > which allows them to be all on the same "LAN". I have attached a simple > diagram illustrating this setup, also available here: > https://i.imgur.com/jc7bc6b.png . I would *prefer* to use the DHCP that > comes "built-in" to livbirt, but I am open to running DHCP on each host, or > as a guest inside each guest network. > > So, I hope this sheds a little more light on my scenario. > If you look at the diagram attached, you can see some *red* lines and > some question marks. These red lines are the only part of the environment > that needs to be configured, everything else seems to work fine; KVM can > provision guests, IPSec tunnel is working, vxlan seems to route/function as > I think I would need it, guests can reach the internet, and guests can ping > everything else inside the environment, *except* guests1 cannot ping > guests2. > > Which makes me question the configuration I have for each guest network. > Each guest network is currently configured with NAT Forwarding, but > because KVMHost1 cannot ping guests on KVMHost2, I am concluding that a > NAT'd guest network is most likely something that I do not want. I have > some choices on how to configure each guest network, it could be an "open > <https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html>" > or "routed <https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode>" > network, but I would still need to configure some static routes. > > I will take a stab at doing an open or routed network today, but in the > meantime, your thoughts and comments about my setup are still welcome and > encouraged. > > Thank you! > -Cobin > > > > On Wed, May 30, 2018 at 10:44 PM Peter Crowther < > peter.crowt...@melandra.com> wrote: > >> I have to say that I wouldn't do the networking that way - in fact, in >> the clusters I manage, we haven't done the networking that way :-). Rather >> than layer 3 routing between VMs, we've chosen to use layer 2 virtual >> switching (yes, using openvswitch). We have the luxury of multiple 10G >> NICs between our hosts, so we've separated out the management network from >> the guest network, simply to ensure that we retain administrative access to >> the hosts via ssh. If you want to live a little more dangerously, you >> could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an >> extra network card on each host for the peace of mind! >> >> For the project that's been live for two years: we presently run four >> hosts on the lab's production network (another two on its acceptance-test >> network, and another one as a "kickaround" host for playing about with >> configs). Guest VMs on all four production hosts share 192.168.59.0/24 >> (why "59" is a story for another day), on an OVS virtual switch on each >> host named br-guest, with the guest-specific NIC also set as a port on the >> virtual switch. Guest traffic is therefore sent transparently between the >> hosts where needed, and we can live-migrate a guest from one host to >> another with no need to change the guest's IP address. Because we share a >> common guest network and IP range between all hosts, it's trivial to add >> (or remove) hosts - no host needs to know anything about routing to another >> host, and in fact only our management layer cares how many hosts we have. >> >> We happen to have "controller" nodes that run redundant DHCP servers with >> non-overlapping scopes, but the exact location is not a requirement of this >> setup. We could equally well set up a DHCP service on the guest network on >> each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host, >> .101 to .200 on another host. Guests will typically receive offers from >> each DHCP server and can choose, which is fine as they're all on the same >> network. This provides redundancy in case of a full or failed DHCP server, >> which your routed network approach wouldn't without some careful DHCP >> forwarding work. >> >> We happen to base our hosts on CentOS 7, but I manage other >> Debian-derived systems and can probably remember enough about its network >> setup to help with Ubuntu. Certainly I can help with OVS weirdnesses; it >> took some time to get my head round exactly how it works. That said, I've >> never set up a kvm host on Debian. >> >> Good luck; happy to provide further pointers if useful. >> >> Cheers, >> >> - Peter >> >> On 30 May 2018 at 15:32, Cobin Bluth <cbl...@gmail.com> wrote: >> >>> Hello Libvirt Users, >>> >>> I would like to setup a two node bare-metal cluster. I need to guidance >>> on the network configuration. I have attached a small diagram, the same >>> diagram can be seen here: https://i.imgur.com/SOk6a6G.png >>> >>> I would like to configure the following details: >>> - Each node has a DHCP enabled guest network where VMs will run. (eg, >>> *192.168.1.0/24 >>> <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24 >>> <http://192.168.2.0/24>* for Host2) >>> - Any guest in Host1 should be able to ping guests in Host2, and vice >>> versa. >>> - All guests have routes to reach the open internet (so that '*yum >>> update*' will work "out-of-the-box") >>> - Each node will be able to operate fully if the other physical node >>> fails. (no central DHCP server, etc) >>> - I would like to *add more physical nodes later* when I need the >>> resources. >>> >>> This is what I have done so far: >>> - Installed latest Ubuntu 18.04, with latest version of libvirt and >>> supporting software from ubuntu's apt repo. >>> - Each node can reach the other via its own eth0. >>> - Each node has a working vxlan0, which can ping the other via its >>> vxlan0, so it looks like the vxlan config is working. (I used *ip link >>> add vxlan0 type vxlan...*) >>> - Configured route on Host1 like so: *ip route add 192.168.2.0/24 >>> <http://192.168.2.0/24> via 172.20.0.1* >>> - Configured route on Host2 also: *ip route add 192.168.1.0/24 >>> <http://192.168.1.0/24> via 172.20.0.2* >>> - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and >>> vice versa, yay. >>> - Guests on Host1 *cannot* ping guests on Host2, I suspect because the >>> the default NAT config of the libvirt network. >>> >>> So, at this point I started to search for tutorials or more >>> information/documentation, but I am a little overwhelmed by the sheer >>> amount of information, as well as a lot of "stale" information on blogs etc. >>> I have learned that I can *virsh net-edit default*, and then change it >>> to an "open" network:* <forward mode='open'/>* >>> After doing this, the guests cannot reach outside their own network, nor >>> reach the internet, so I assume that I would need to add some routes, or >>> something else to get the network functioning like I want it. There is also >>> *<forward >>> mode="route"/>*, but I dont fully understand the scenarios where one >>> would need an *open* or a *route* forward mode. I have also shied away >>> from using openvswitch, and have opted for ifupdown2. >>> (I have taken most of my inspiration from this blog post: >>> https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/ >>> ) >>> >>> Some questions that I have for the mailing list, any help would be >>> greatly appreciated: >>> - Is my target configuration of a KVM cluster uncommon? Do you see >>> drawbacks of this setup, or does it go against "typical convention"? >>> - Would my scenario be better suited for an "*open*" network or a " >>> *route*" network? >>> - What would be the approach to complete this setup? >>> >>> >>> >>> >>> _______________________________________________ >>> libvirt-users mailing list >>> libvirt-users@redhat.com >>> https://www.redhat.com/mailman/listinfo/libvirt-users >>> >> >>
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
