Hi Ryan,
On Thu, May 19, 2016 at 3:51 PM, Ryan Moats <rmo...@us.ibm.com> wrote:
>
> As I mentioned at today's IRC meeting - we have a need to include
measuring
> the performance of the OVN control channel under scale as part of our
CI/CD
> test suite, which is driving us to examine the feasibility of some sort of
> lightweight instrumentation of ovsdb-server, ovn-controller, and
ovn-northd.
> While I'm familiar with the counters that are dumped during an excessive
poll
> cycle, those don't entirely meet our needs, as I am looking for something
> that allows for the amount of clock time spent in each method as our
> starting point.
>
> The reason for the "lightweight" adjective above is that we are looking to
> run the same code base in both CI/CD and production, and that means
attaching
> a profiler to the code base is contra-indicated. My initial thought is to
> create a new general arg (--profile) that sets a boolean (for example
> is_profiling), then define a macro like the following:
>
> #define INSTRUMENT(x) (if (is_profiling) { VLOG_DEBUG(x); })
>
> and then decorating methods with INSTRUMENT at its start and exit.
>
> But, this is an awful lot of decorating and visual noise and so before
pushing
> a large patch series, I wanted to throw the idea on the table and let
people
> pick it apart.
>
> Ryan (regXboi)
>
>
> _______________________________________________
> discuss mailing list
> discuss@openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
>
Are you suggesting to log some timestamps with this macro so that we can
get the time spent in the functions we are interested in? I have no
objection to this approach. It seems to me not contradict with the counter
approach. Since the functions we are going to profile themselves are
already very heavy (e.g. physical_run(), lflow_run(), ...), adding a
counter would not have any real impact even when running in production. The
limitation of counters is that it cannot tell the time spent in a specific
function directly, but only tells the total number and the avg speed of the
whole execution flow. So these 2 approaches may be combined to evaluate the
performance.
At the same time we want to measure the end-to-end c-plane performance
accurately (despite of code logic/structure). Current implementation in
ovs-scale-test is just to reflect the end-to-end cost by checking the time
between port creation and the port state coming up in NB.
While it reflects the cost indirectly, it is not the actual time used for
the end-to-end flow. There are 2 execution flows involved:
1. Port-create -> NB ovsdb -> northd -> SB ovsdb (lflow)
2. Port-bind -> local ovsdb -> ovn-controller -> SB ovsdb -> northd -> NB
ovsdb (port state up) -> other ovn-controllers -> ovs open-flow table (data
plane ready)
For the 2nd flow, to measure the time accurately we want to check the flows
in ovs for all the HVs that are supposed to have the connectivity of that
port. One way to do that is polling with command:
ovs-ofctl dump-flows … mac=<mac of the newly binded port>
However, polling for the flow readyness seems too heavy. It would be better
to get notified when an expected flow is programmed in OVS flow table.
Would it be feasible to implement such a subscription and notification
mechanism for ovs flow table?
Han
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
