Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21653
Change subject: IMPALA-12737: Limit workload management initialization to one coordinator. ...................................................................... IMPALA-12737: Limit workload management initialization to one coordinator. The workload management initialization process creates the two tables "sys.impala_query_log" and "sys.impala_query_live" during coordinator startup. The current design for this init process is to create both tables on each coordinator at startup by running create table if not exists DDLs. This design causes unnecessary DDLs to execute which delays coordinator startup and introduces the potential for unnecessary startup failures should the DDLs fail. Additionally, this design will not work once table schema modifications are needed since multiple coordinators running the same create and alter table statements while the catalog is warming results in conflicts. This change introduces using the statestore to determine which coordinator will run the workload management initialization process. The first coordinator that puts a message on the new impala-workload-management statestore topic will be the coordinator that runs all the DDLs to create the workload management tables. This coordinator places a follow-up message on the same statestore topic to indicate it has completed workload management initialization. At that point, all the coordinators start writing completed queries to the completed queries table. Additionally, this patch adds functionality to first check if the two workload management tables already exist and skip running the create table DDL if they already exist. A new coordinator startup flag "workload_mgmt_schema_version" has been added to prepare for changing the workload management tables schemas. Testing is accomplished by adding new ctests and custom cluster tests. The workload management init process is tested by starting a cluster, adding more coordinators after the cluster completes startup, and forcing delays in the startup process to simulate delays in forming a large cluster. Change-Id: Id645f94c8da73b91c13a23d7ac0ea026425f0f96 --- M be/src/service/CMakeLists.txt M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/workload-management-fields.cc M be/src/service/workload-management-flags.cc A be/src/service/workload-management-init.cc A be/src/service/workload-management-init.h M be/src/service/workload-management.cc M be/src/service/workload-management.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/CMakeLists.txt A be/src/util/version-util-test.cc A be/src/util/version-util.cc A be/src/util/version-util.h M bin/start-impala-cluster.py M tests/common/impala_cluster.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_query_live.py M tests/custom_cluster/test_query_log.py A tests/custom_cluster/test_workload_mgmt_init.py 21 files changed, 1,227 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21653/3 -- To view, visit http://gerrit.cloudera.org:8080/21653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id645f94c8da73b91c13a23d7ac0ea026425f0f96 Gerrit-Change-Number: 21653 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
