Hello Jason Fehr, Pranav Lodha, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23588
to look at the new patch set (#8).
Change subject: IMPALA-9715: Add devdata dataload option
......................................................................
IMPALA-9715: Add devdata dataload option
Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.
Enable with `./buildall.sh -devdata`. Also skips loading tables that
require Hive as they're less commonly used in testing.
Data load with this option (first 3 run in parallel)
Loading TPC-H data OK (Took: 0 min 15 sec)
Loading TPC-DS data OK (Took: 2 min 18 sec)
Loading functional-query data OK (Took: 3 min 43 sec)
Loading Hive UDFs OK (Took: 0 min 12 sec)
Running custom post-load steps OK (Took: 0 min 7 sec)
Caching test tables OK (Took: 0 min 3 sec)
Creating internal HBase table OK (Took: 0 min 12 sec)
Checking HDFS health OK (Took: 0 min 2 sec)
Logging created files OK (Took: 0 min 2 sec)
Loading external data sources OK (Took: 0 min 19 sec)
Computing table stats OK (Took: 0 min 14 sec)
Creating tpcds testcase data OK (Took: 0 min 47 sec)
Reduces dataload from ~30 minutes to ~5 minutes with 16 cores.
Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M bin/load-data.py
M buildall.sh
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
5 files changed, 72 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/8
--
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Pranav Lodha <[email protected]>
