Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/22778 )
Change subject: IMPALA-13960: Add catalog timeline item for
prepareInsertEventData
......................................................................
IMPALA-13960: Add catalog timeline item for prepareInsertEventData
When enable_insert_events is set to true (default), Impala will fire HMS
INSERT events for each INSERT statement. Preparing data of the
InsertEvents actually takes time since it fetches checksums of all the
new files. This patch adds a catalog timeline item to reveal this step.
Before this patch, the duration of "Got Metastore client" before "Fired
Metastore events" could be long:
Catalog Server Operation: 65.762ms
- Got catalog version read lock: 12.724us (12.724us)
- Got catalog version write lock and table write lock: 224.572us
(211.848us)
- Got Metastore client: 418.346us (193.774us)
- Got Metastore client: 29.001ms (28.583ms) <---- Unexpected long
- Fired Metastore events: 52.665ms (23.663ms)
After this patch, we shows what actually takes the time is "Prepared
InsertEvent data":
Catalog Server Operation: 61.597ms
- Got catalog version read lock: 7.129us (7.129us)
- Got catalog version write lock and table write lock: 114.476us
(107.347us)
- Got Metastore client: 200.040us (85.564us)
- Prepared InsertEvent data: 25.335ms (25.135ms)
- Got Metastore client: 25.342ms (7.009us)
- Fired Metastore events: 46.625ms (21.283ms)
Tests:
- Added e2e test
Change-Id: Iaef1cae7e8ca1c350faae8666ab1369717736978
Reviewed-on: http://gerrit.cloudera.org:8080/22778
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/query_test/test_observability.py
3 files changed, 13 insertions(+), 4 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/22778
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaef1cae7e8ca1c350faae8666ab1369717736978
Gerrit-Change-Number: 22778
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
