Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22061
Change subject: IMPALA-13543: single_node_perf_run.py must accept tpcds_partitioned ...................................................................... IMPALA-13543: single_node_perf_run.py must accept tpcds_partitioned tpcds_partitioned dataset is a fully-partitioned version of tpcds dataset (the latter only partition store_sales table). It does not have the default text format database like tpcds dataset. Instead, it relies on pre-existence of text format tpcds database, which then INSERT OVERWRITE INTO tpcds_partitioned database equivalent. It does not have its own queries set, but instead symlinked to share testdata/workloads/tpcds/queries. It also have slightly different schema from tpcds dataset, namely column "c_last_review_date" in tpcds dataset is "c_last_review_date_sk" in tpcds_partitioned (TPC-DS v2.11.0, see related commit in impala-tpcds-kit). These reasons make tpcds_partitioned ineligible for perf-AB-test (single_node_perf_run.py) This patch update single_node_perf_run.py and related scripts to make tpcds_partitioned eligible for benchmark dataset. It adds an initial steps to load the text database from tpcds dataset with selected scale before running the load script for tpcds_partitioned dataset. Compute stats step also limited to run one at a time to not overadmit the cluster with concurrent compute stats queries. Testing - Run perf-AB-test-ub2004 with this commit included and confirm benchmark works with tpcds_partitioned dataset. Change-Id: I4b6f435705dcf873696ffd151052ebeab35d9898 --- M bin/single_node_perf_run.py M testdata/bin/generate-schema-statements.py M testdata/datasets/tpcds_partitioned/tpcds_partitioned_schema_template.sql M tests/util/test_file_parser.py 4 files changed, 118 insertions(+), 77 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/22061/1 -- To view, visit http://gerrit.cloudera.org:8080/22061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4b6f435705dcf873696ffd151052ebeab35d9898 Gerrit-Change-Number: 22061 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
