Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22061 )
Change subject: IMPALA-13543: single_node_perf_run.py must accept tpcds_partitioned ...................................................................... IMPALA-13543: single_node_perf_run.py must accept tpcds_partitioned tpcds_partitioned dataset is a fully-partitioned version of tpcds dataset (the latter only partition store_sales table). It does not have the default text format database like tpcds dataset. Instead, it relies on pre-existence of text format tpcds database, which then INSERT OVERWRITE INTO tpcds_partitioned database equivalent. It does not have its own queries set, but instead symlinked to share testdata/workloads/tpcds/queries. It also have slightly different schema from tpcds dataset, namely column "c_last_review_date" in tpcds dataset is "c_last_review_date_sk" in tpcds_partitioned (TPC-DS v2.11.0, section 2.4.7). These reasons make tpcds_partitioned ineligible for perf-AB-test (single_node_perf_run.py). This patch update single_node_perf_run.py and related scripts to make tpcds_partitioned eligible for benchmark dataset. It adds an initial steps to load the text database from tpcds dataset with selected scale before running the load script for tpcds_partitioned dataset. Compute stats step also limited to run one at a time to not overadmit the cluster with concurrent compute stats queries. Created helper function build_replacement_params() inside generate-schema-statements.py for common function. Testing - Run perf-AB-test-ub2004 with this commit included and confirm benchmark works with tpcds_partitioned dataset. - Run normal data loading. Pass FE tests, and query_test/test_tpcds_queries.py. Change-Id: I4b6f435705dcf873696ffd151052ebeab35d9898 Reviewed-on: http://gerrit.cloudera.org:8080/22061 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M bin/single_node_perf_run.py M testdata/bin/generate-schema-statements.py M testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql M tests/util/test_file_parser.py 4 files changed, 184 insertions(+), 111 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/22061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4b6f435705dcf873696ffd151052ebeab35d9898 Gerrit-Change-Number: 22061 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
