Hi On 20 September 2012 04:01, Stephen Hemminger <shemmin...@vyatta.com> wrote: > On Wed, 19 Sep 2012 13:57:39 -0400 > christopher barry <cba...@rjmetrics.com> wrote: > >> On Tue, 2012-09-18 at 23:49 -0400, christopher barry wrote: >> > if this is not the best list for user questions, please send me to >> > wherever the best place is for that. >> > >> > I'm building an opennebula cloud, and I have openvswitch master running >> > on squeeze kvm host nodes with 3.5.4 as bridge br0 with a single port >> > (eth0 at the moment). Each host node also has a Mellanox QDR HCA for >> > storage connectivity. I've configured an ipoib interface (ib0) as well >> > on each node. eth0 and ib0 are on separate subnets. >> > >> > Thinking it would be faster to use ib0 as the bridge port, rather than >> > eth0, I reconfigured all the bridges in /etc/network/interfaces to use >> > ib0, removing eth0 as a port in the file, and bounced the cluster. >> > >> > couple of things happened: >> > * ib0 did get added as a port, but eth0 remained unexpectedly (I had to >> > del-port it) (that may be by design though, dunno) >> > * I could not pass traffic through br0 when ib0 was a port. >> > * I could ping the bridge with ib0 from it's owning host, but not from >> > outside. (assuming it's just br0 answering, and ib0 is non-functional) >> > >> > I'm confused about what is happening. By itself, the ib0 is functioning >> > fine, and the bridge works fine with eth0. Is it because ib0 is not a >> > real ethernet nic, per se, that I can't make this work? Was it dumb to >> > even try it? Or, am I just missing something obvious? >> > >> > Thanks for any insight, >> > -C >> > >> >> No takers? > > Isn't Infiniband and Ethernet addressing different? > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
Your issue here is ib0 is an IPoIB device, it can only pass IP packets. OVS is a Layer 2 virtual switch (similar to the linux bridge module PS: Thanks Stephen. :P) that only forwards Ethernet frames. This leaves you with a few options, listed in basically order of difficulty and closeness to Ethernet. Almost all of them require writing your own network hooks for OpenNebula. a) Use routed interfaces b) Build a brouter (look up ebtables/iptables for how to do this) c) Tunnel Ethernet over IP (using GRE, L2TP etc) and bridge that with OVS d) Use Mellanox BridgeX devices + the eoib module from Mellanox OFED package d) Implement your own Ethernet on IB encapsulation Sadly there is currently not much OSS work going on to support this and the state of converged Infiniband clouds is really hurting for it. Mellanox has proposed an eIPoIB device driver to LKML but in it's current state/design it has no chance of being merged. Joseph. -- CTO | Orion Virtualisation Solutions | www.orionvm.com.au Phone: 1300 56 99 52 | Mobile: 0428 754 846 _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
