On Tue, Apr 20, 2010 at 11:22 AM, Trygve Sanne Hardersen < try...@hypobytes.com> wrote: > > I'm running XCP on 2 physical machines configured as a pool. Each box > has two normal Ethernet NICs (with 6-byte HW addresses) and a single > Infiniband NIC (with a 20-byte HW address). > > My original plan was to run the inter-vm network over the IB link, but > this requires the OVS or Linux bridge to work. Instead I'm now running > the XCP management interface on the IB link. This makes XCP create a > bridge interface "brib0" with a broken OVS connection to "ib0". When I > manually configure ib0 and route traffic through this interface using > the Linux routing table communication works and I have very fast > communication between the hosts in the pool. >
Getting the VMs in a bridge with an Infiniband device requires more than just supporting multiple MAC address lengths. The VMs are speaking Ethernet and what you want would require bidirectional translation between Ethernet and Infiniband protocols. Ethernet and Infiniband may appear to be the same to an application because in this case they are both running IP and the application is using IP or some protocol on top of IP. However, at the L2 protocol level they are not the same and any piece of software that operates at L2, such as OVS or the virtual Ethernet devices of the guest VMs, will not just work. > > The problem I'm now facing is that XCP runs the VM consoles on the > brib0 interface but I'm unable to route traffic from one of the > Ethernet NICs to this interface. I've tried to do this using iptables > but it seems the lib_REDIRECT kernel module is unavailable on XCP. The > simplest solution would probably be to run the XCP consoles on a > different NIC than the management interface but I've been unable to > configure this so far. Routing traffic to the interface is probably > also possible but I've not achieved this yet. > Questions about configuring XCP are really better suited to their mailing lists, though I wouldn't be surprised if dividing up the management interface in the way that you are looking for isn't supported. Personally, I wouldn't try to put the Infiniband device in a bridge at all - it won't work so there isn't much point. Just configure the device manually, as Ben suggested. You can create static routing entries in either dom0 or the domUs so that the traffic you do want to do go over the Infiniband device is directed there.
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss_openvswitch.org
