Hi Alessandro, Resending my last email since the formatting went off for some reason.
Hi Saurabh, Thanks for posting these patches! We looked at the code in the last two days, running various tests and did comparisons with our porting. We tried to be as unbiased as possible in the review, with the goal of making the best out of both codebases and not a “us” vs “them” match. :) For the sake of simplicity I’ll refer to these patches as “VMWare” and to our patches as “Cloudbase” (available at [10]) in the rest of the text. General impressions (VMWare patches) The code is well written and easy to follow, which helped in the analysis. In general, at the current stage the patches look more like a proof of concept or an early work in progress and we experienced various blue screens (kernel panics) when using the compiled kernel extension while performing basic actions. Estimating the amount of time required to reach a maturity level comparable with the Cloudbase porting goes outside of the scope of this initial analysis, but it’s surely non negligible. Thanks for taking a look at the code. I am glad that you were able to follow the code with ease. We have analyzed the crashes/errors that you reported. I discuss them later in their relevant sections. With regards to your codebase, we have been waiting to see your patches to the tip of openvswitch repository so that we can make an apples-to-apples comparison of the two code bases. Feature comparison Here's an initial raw feature comparison, please send any correction in case anything is missing. Feature | Cloudbase | VMWare | --------------------------------------------------------- VXLAN | X | X | VXLAN multiple vports | X | | VXLAN configurable UDP ports | X | | GRE | X | | IPv4 packet fragmentation | X | | LLC | | X | VLAN 802.1Q | X | X | Megaflows | X | | Multiple Hyper-V switches | X | | --------------------------------------------------------- We have intentionally chosen not to publish everything with this first version. Several features are almost ready or are actively being developed and will be published soon. For example – megaflows, Geneve tunnels & ovs-wind. Few questions/clarifications: - Can you explain what you mean by “VXLAN multiple vports”? - We do support IPv4 fragmentation on the Tx side. If the encapsulated packet is larger than MTU, we will fragment it. In addition, we also support handling fragmented IPv4 packets on the Rx side. - Megaflows is an active work-in-progress and patches for the same will be posted soon. - Can you explain the need for having “Multiple Hyper-V switches”? Also, is this something that you support right now? Both implementations are limited to one single datapath at the moment. We have a WiP patch for multiple datapaths, but not yet public. #### General design #### Both porting efforts consists in an Hyper-V extensible switch NDIS forwarding extension. In both cases, external switches require //AllowManagementOS// set to true in order to manage tunnel endpoint traffic. This will change in future releases. What is a use case for this? We see a lot of overheads in trying to manage tunnel endpoints outside the host stack and are not planing to support this. Compatible platforms: Hyper-V Server / Windows Server 2012 and 2012 R2 and Windows 8 / 8.1 There are some important design differences between the two portings: ##### Interaction with userspace ###### Windows does not have a Netlink socket based equivalent, which is where all the communication between the user and kernel code is performed (current master branch). Cloudbase ========= We opted for the minimum impact in the userspace code, to minimize the code paths and maintenance required for the various platforms. To obtain this result, a Netlink equivalent has been created providing functions with the same semantic and an implementation that fits the Windows mode. Please see [1] for details. Advantages * Minimal impact in the current codebase (reduced maintenance) * Another advantage is that adapting the porting to multiple OVS versions is easy. * Command line compatibility Disadvantages * This design requires some additional marshalling between the userspace and kernel data structures, but from a first set of tests the performance impact is not relevant. VMWare ====== The patches contain separate Windows specific userspace functions, mostly contained in "dpif-windows.c" and "netdev-windows.c". The "DESIGN" file contains a reference to an additional component that possibly replaces ovs-vswitchd on Windows called ovs-wind, NOT present in the patches. Advantages * The main advantage consists in having less traffic between the components Disadvantages * Non trivial amount of code to be maintained in userspace. * Command line compatibility (?) This will probably boil down to what the OVS developers are more comfortable with. There are pros/cons either ways. Few comments: 01. We believe that our implementation of user to kernel mode interface is aligned with the OVS design where the DPIF provisioning layer is platform specific. It is confusing/non-obvious to have dpif-linux talk to windows. It took us a while to figure it out ourselves. It probably also muddles more files in userspace with ugly #ifdefs, which tend to make the code less readable. Perhaps, we can get a better idea once you post all your userspace changes. 02. We don’t feel that compatibility between all user/kernel versions is that useful for the windows platform. The kernel & user space will usually go hand-in-hand. It keeps things simple, easy to maintain, easy to test & easy to debug. However, this restriction can potentially be overcome. 03. Implementing a Windows specific datapath interface should not break command line compatibility for any of the daemons or control commands. If it does, they will be fixed. ##### Hyper-V / OVS port matching ##### Cloudbase ========= Ports are created independently on the OVS and Hyper-V switches and matched with the //Set-VMNetworkAdapterOVSPort// Powershell cmdlet. See [2] for details. Advantages * Flexibility: OVS ports can have any name and can be created independently of attached VMs. * Compatibility with scripts using //ovs-vsctl// Disantages * One additional configuration step is required when a VM is attached to a switch VMWare ====== OVS ports are automatically created when a Hyper-V port is created on the switch. Ports have random names that also change between reboots, see [3]. There's no standard and easy way to determine which OVS port corresponds to a Hyper-V port. This can be improved by using the Hyper-V port GUID. Advantages * Less configuration Disadvantages * Determine which VM is connected to what port is not currently possible. * Security considerations: what OpenFlow rules are applied when a port is created? Yes, most of the VMWare restrictions will go away with the ‘ovs-wind’ implementation. The idea is to reduce configuration overheads & simplify VM port management. Ovs-wind is actively being developed and will be released soon. ##### Stability ##### Cloudbase ========= The driver is already in beta status and available for download [8]. A Jenkins job builds both userspace and kernel components at every build, the driver is signed and an installer (external) is provided for an easy deployment (both GUI and unattended). Stability is compatible with a beta release. A few issues need to be solved and performance can be improved as detailed in [8]. It'd be great if somebody in the community could do some testing and provide a separate opinion. VMWare ====== As noted before, based on the tests that we managed to perform the driver seems to be in an early development stage. We're definitely open to have a discussion on IRC for setting up an environment that allows more testing, as the crashes limit the actions that can be performed. Examples of issues we encountered: * Enabling the driver on a plain vanilla Hyper-V 2012 R2 having a single external switch with AllowManagementOS set, panics (blue screen) with the error reported at [4] (WInDbg, !analyze -v). Co-incidentally, we hit the same crash recently on one of our new setups. The root cause has been identified and the fix has also been tested. We will include the fix in our V2 patch which will be published fairly soon. * In the same OS, but with AllowManagementOS not set, it crashes with the error visible in [5]. This is an unsupported configuration and goes against the guidelines of using our extension. Having said that, the extension should not crash. We will work on making sure that it doesn’t. * ovs-vsctl.exe --db=tcp:127.0.0.1:8888 add-br br0 fails with [6] * ovs-vsctl.exe --db=tcp:127.0.0.1:8888 show issues [7] These errors are harmless, you should still be able to configure the switch. * ovs-ofctl.exe benchmark / dump-ports-desc inconsistencies [11] This inconsistency is caused by a bug when calling xgettimeofday() directly. Guru has posted a patch for fix this upstream - http://patchwork.openvswitch.org/patch/4808/. ##### Performance ##### Due to the issues encountered during basic testing it was not possible to perform a proper performance comparison. ##### Notes ##### * The VMWare patches employ also Windows Filtering Platform (WFP) features for IP packet management, while the Cloudbase one performs those operations with ad hoc functions. * The Cloudbase porting must be rebased to the tip of the master branch (currently this requires around 1 week). ##### Conclusions ##### This is just a first quick comparison between both porting efforts including design, features and stability. A follow up conversation will also include notes on the implementation. Since the objective is to compare the two porting efforts and merge the best parts of each one, I'd like to propose a follow up via IRC, Hangout, Skype, etc. to discuss the details among the two teams. My IRC nick is: alexpilotti on FreeNode Thanks! Alessandro Thanks! Saurabh _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
