Re: [libcxx] r278147 - Update in-tree Google Benchmark to current ToT.

Craig, Ben via cfe-commits Tue, 09 Aug 2016 12:22:35 -0700

This is probably more of a comment on the compare_bench.py side ofthings, but here goes anyway...

When comparing two results, both the percentage and absolute differencescan be very important. It is very easy to be misled by percentages. Itis harder to be misled with absolute differences.

For example, I think it is more insightful to see something like thisfor the comparison:


Comparing ./util_smartptr.native.out to ./util_smartptr.libcxx.out
Benchmark                          Time           CPU
-----------------------------------------------------
BM_SharedPtrCreateDestroy         -16 ns (-26%) -16 ns (-26%)
BM_SharedPtrIncDecRef             +0 ns (+0%)   +0 ns (+0%)
BM_WeakPtrIncDecRef               +6 ns (+21%)  +6 ns (+21%)

This presentation makes it easier to evaluate the significance ofshifted costs. The shared_ptr changes moved some costs around, and thisbenchmark shows that 16 ns left one area, and turned into 6 nselsewhere. The percentages mask that insight.

I've been in benchmarking situations where one function's time isreduced by 5%, and another function increases by 100%. Some managersstarted panicking because of it. In actuality, 5 ms was shaved off of a100 ms operation that was in the inner loop. The 5 ms was moved intothe outer loop which previously took 5 ms.



On 8/9/2016 1:56 PM, Eric Fiselier via cfe-commits wrote:

Author: ericwf
Date: Tue Aug  9 13:56:48 2016
New Revision: 278147

URL: http://llvm.org/viewvc/llvm-project?rev=278147&view=rev
Log:
Update in-tree Google Benchmark to current ToT.

I've put some work into the Google Benchmark library in order to make it easier
to benchmark libc++. These changes have already been upstreamed into
Google Benchmark and this patch applies the changes to the in-tree version.

The main improvement in the addition of a 'compare_bench.py' script which
makes it very easy to compare benchmarks. For example to compare the native
STL to libc++ you would run:

`$ compare_bench.py ./util_smartptr.native.out ./util_smartptr.libcxx.out`

And the output would look like:

RUNNING: ./util_smartptr.native.out
Benchmark                          Time           CPU Iterations
----------------------------------------------------------------
BM_SharedPtrCreateDestroy         62 ns         62 ns   10937500
BM_SharedPtrIncDecRef             31 ns         31 ns   23972603
BM_WeakPtrIncDecRef               28 ns         28 ns   23648649
RUNNING: ./util_smartptr.libcxx.out
Benchmark                          Time           CPU Iterations
----------------------------------------------------------------
BM_SharedPtrCreateDestroy         46 ns         46 ns   14957265
BM_SharedPtrIncDecRef             31 ns         31 ns   22435897
BM_WeakPtrIncDecRef               34 ns         34 ns   21084337
Comparing ./util_smartptr.native.out to ./util_smartptr.libcxx.out
Benchmark                          Time           CPU
-----------------------------------------------------
BM_SharedPtrCreateDestroy         -0.26         -0.26
BM_SharedPtrIncDecRef             +0.00         +0.00
BM_WeakPtrIncDecRef               +0.21         +0.21

Added:
     libcxx/trunk/utils/google-benchmark/test/multiple_ranges_test.cc
     libcxx/trunk/utils/google-benchmark/test/register_benchmark_test.cc
     libcxx/trunk/utils/google-benchmark/tools/
     libcxx/trunk/utils/google-benchmark/tools/compare_bench.py
     libcxx/trunk/utils/google-benchmark/tools/gbench/
     libcxx/trunk/utils/google-benchmark/tools/gbench/Inputs/
     libcxx/trunk/utils/google-benchmark/tools/gbench/Inputs/test1_run1.json
     libcxx/trunk/utils/google-benchmark/tools/gbench/Inputs/test1_run2.json
     libcxx/trunk/utils/google-benchmark/tools/gbench/__init__.py
     libcxx/trunk/utils/google-benchmark/tools/gbench/report.py
     libcxx/trunk/utils/google-benchmark/tools/gbench/util.py
Modified:
     libcxx/trunk/benchmarks/CMakeLists.txt
     libcxx/trunk/benchmarks/ContainerBenchmarks.hpp
     libcxx/trunk/benchmarks/algorithms.bench.cpp
     libcxx/trunk/benchmarks/unordered_set_operations.bench.cpp
     libcxx/trunk/utils/google-benchmark/AUTHORS
     libcxx/trunk/utils/google-benchmark/README.md
     libcxx/trunk/utils/google-benchmark/include/benchmark/benchmark_api.h
     libcxx/trunk/utils/google-benchmark/include/benchmark/macros.h
     libcxx/trunk/utils/google-benchmark/include/benchmark/reporter.h
     libcxx/trunk/utils/google-benchmark/src/benchmark.cc
     libcxx/trunk/utils/google-benchmark/src/colorprint.cc
     libcxx/trunk/utils/google-benchmark/src/colorprint.h
     libcxx/trunk/utils/google-benchmark/src/complexity.cc
     libcxx/trunk/utils/google-benchmark/src/console_reporter.cc
     libcxx/trunk/utils/google-benchmark/src/cycleclock.h
     libcxx/trunk/utils/google-benchmark/src/sysinfo.cc
     libcxx/trunk/utils/google-benchmark/test/CMakeLists.txt
     libcxx/trunk/utils/google-benchmark/test/basic_test.cc
     libcxx/trunk/utils/google-benchmark/test/benchmark_test.cc
     libcxx/trunk/utils/google-benchmark/test/complexity_test.cc
     libcxx/trunk/utils/google-benchmark/test/fixture_test.cc
     libcxx/trunk/utils/google-benchmark/test/map_test.cc
     libcxx/trunk/utils/google-benchmark/test/options_test.cc
     libcxx/trunk/utils/google-benchmark/test/reporter_output_test.cc
     libcxx/trunk/utils/google-benchmark/test/skip_with_error_test.cc

Modified: libcxx/trunk/benchmarks/CMakeLists.txt
URL: 
http://llvm.org/viewvc/llvm-project/libcxx/trunk/benchmarks/CMakeLists.txt?rev=278147&r1=278146&r2=278147&view=diff
==============================================================================
--- libcxx/trunk/benchmarks/CMakeLists.txt (original)
+++ libcxx/trunk/benchmarks/CMakeLists.txt Tue Aug  9 13:56:48 2016
@@ -87,7 +87,12 @@ macro(add_benchmark_test name source_fil
    add_executable(${libcxx_target} EXCLUDE_FROM_ALL ${source_file})
    add_dependencies(${libcxx_target} cxx google-benchmark-libcxx)
    add_dependencies(libcxx-benchmarks ${libcxx_target})
-  target_link_libraries(${libcxx_target} cxx -lbenchmark)
+  if (LIBCXX_ENABLE_SHARED)
+    target_link_libraries(${libcxx_target} cxx_shared)
+  else()
+    target_link_libraries(${libcxx_target} cxx_static)
+  endif()
+  target_link_libraries(${libcxx_target} -lbenchmark)
    set_target_properties(${libcxx_target}
      PROPERTIES
            OUTPUT_NAME "${name}.libcxx.out"

Modified: libcxx/trunk/benchmarks/ContainerBenchmarks.hpp
URL: 
http://llvm.org/viewvc/llvm-project/libcxx/trunk/benchmarks/ContainerBenchmarks.hpp?rev=278147&r1=278146&r2=278147&view=diff
==============================================================================
--- libcxx/trunk/benchmarks/ContainerBenchmarks.hpp (original)
+++ libcxx/trunk/benchmarks/ContainerBenchmarks.hpp Tue Aug  9 13:56:48 2016
@@ -10,7 +10,7 @@ namespace ContainerBenchmarks {