Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20377 )
Change subject: IMPALA-12385: Enable Periodic metrics by default ...................................................................... IMPALA-12385: Enable Periodic metrics by default This patch enables periodic metrics in query profiles by default and changes the metric collectors to be more suitable for mixed workloads. -Changed default of resource_trace_ratio to 1. -Changed profile metrics to use sampling counters which can automatically resize for long queries. -Reduced profile metric samping interval to 50ms to support short-running queries. -Changed fragment metrics to use the same sampling interval as periodic metrics. -Added singleton PeriodicCounterUpdater and thread for updating system (KRPC) metrics at a different period than fragment metrics. -Added new flag periodic_system_counter_update_period_ms for configuring system metric update period. Default is 500ms. Example profile output: - HostCpuIoWaitPercentage (400.000ms): 1, 0, 2, 3, 4, 6, 5, 2, 1, ... - HostCpuSysPercentage (400.000ms): 1, 12, 10, 11, 11, 5, 3, 3, 3, ... - HostCpuUserPercentage (400.000ms): 8, 46, 39, 39, 39, 29, 22, 23, ... Testing: -Updated runtime-profile-test and test_observability.py for new defaults -Manual inspection of query profile metrics for long-running queries -Tested for performance regression using 50 concurrent TPCH Q1 and with periodic_counter_update_period_ms=1 to verify that the periodic update loop does not affect peformance or become a bottleneck Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Reviewed-on: http://gerrit.cloudera.org:8080/20377 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Michael Smith <[email protected]> --- M be/src/runtime/exec-env.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/query-state.cc M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/streaming-sampler.h M common/thrift/Query.thrift M tests/query_test/test_observability.py 12 files changed, 147 insertions(+), 81 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 11 Gerrit-Owner: Kurt Deschler <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]>
