On Thu, 2012-09-20 at 05:51 +1000, Joseph Glanville wrote: > Hi > > On 20 September 2012 04:01, Stephen Hemminger <shemmin...@vyatta.com> wrote: > > On Wed, 19 Sep 2012 13:57:39 -0400 > > christopher barry <cba...@rjmetrics.com> wrote: > > > >> On Tue, 2012-09-18 at 23:49 -0400, christopher barry wrote: > >> > if this is not the best list for user questions, please send me to > >> > wherever the best place is for that. > >> > > >> > I'm building an opennebula cloud, and I have openvswitch master running > >> > on squeeze kvm host nodes with 3.5.4 as bridge br0 with a single port > >> > (eth0 at the moment). Each host node also has a Mellanox QDR HCA for > >> > storage connectivity. I've configured an ipoib interface (ib0) as well > >> > on each node. eth0 and ib0 are on separate subnets. > >> > > >> > Thinking it would be faster to use ib0 as the bridge port, rather than > >> > eth0, I reconfigured all the bridges in /etc/network/interfaces to use > >> > ib0, removing eth0 as a port in the file, and bounced the cluster. > >> > > >> > couple of things happened: > >> > * ib0 did get added as a port, but eth0 remained unexpectedly (I had to > >> > del-port it) (that may be by design though, dunno) > >> > * I could not pass traffic through br0 when ib0 was a port. > >> > * I could ping the bridge with ib0 from it's owning host, but not from > >> > outside. (assuming it's just br0 answering, and ib0 is non-functional) > >> > > >> > I'm confused about what is happening. By itself, the ib0 is functioning > >> > fine, and the bridge works fine with eth0. Is it because ib0 is not a > >> > real ethernet nic, per se, that I can't make this work? Was it dumb to > >> > even try it? Or, am I just missing something obvious? > >> > > >> > Thanks for any insight, > >> > -C > >> > > >> > >> No takers? > > > > Isn't Infiniband and Ethernet addressing different? > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > http://openvswitch.org/mailman/listinfo/dev > > Your issue here is ib0 is an IPoIB device, it can only pass IP packets. > OVS is a Layer 2 virtual switch (similar to the linux bridge module > PS: Thanks Stephen. :P) that only forwards Ethernet frames. > > This leaves you with a few options, listed in basically order of > difficulty and closeness to Ethernet. > Almost all of them require writing your own network hooks for OpenNebula. > > a) Use routed interfaces > b) Build a brouter (look up ebtables/iptables for how to do this) > c) Tunnel Ethernet over IP (using GRE, L2TP etc) and bridge that with OVS > d) Use Mellanox BridgeX devices + the eoib module from Mellanox OFED package > d) Implement your own Ethernet on IB encapsulation > > Sadly there is currently not much OSS work going on to support this > and the state of converged Infiniband clouds is really hurting for it. > Mellanox has proposed an eIPoIB device driver to LKML but in it's > current state/design it has no chance of being merged. > > Joseph. >
Excellent explanation. Thank You. -C _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
