Quanlong Huang has uploaded this change for review. (
http://gerrit.cloudera.org:8080/22778
Change subject: IMPALA-13960: Add catalog timeline item for
prepareInsertEventData
......................................................................
IMPALA-13960: Add catalog timeline item for prepareInsertEventData
When enable_insert_events is set to true (default), Impala will fire HMS
INSERT events for each INSERT statement. Preparing data of the
InsertEvents actually takes time since it fetches checksums of all the
new files. This patch adds a catalog timeline item to reveal this step.
Before this patch, the duration of "Got Metastore client" before "Fired
Metastore events" could be long:
Catalog Server Operation: 65.762ms
- Got catalog version read lock: 12.724us (12.724us)
- Got catalog version write lock and table write lock: 224.572us
(211.848us)
- Got Metastore client: 418.346us (193.774us)
- Got Metastore client: 29.001ms (28.583ms) <---- Unexpected long
- Fired Metastore events: 52.665ms (23.663ms)
After this patch, we shows what actually takes the time is "Prepared
InsertEvent data":
Catalog Server Operation: 61.597ms
- Got catalog version read lock: 7.129us (7.129us)
- Got catalog version write lock and table write lock: 114.476us
(107.347us)
- Got Metastore client: 200.040us (85.564us)
- Prepared InsertEvent data: 25.335ms (25.135ms)
- Got Metastore client: 25.342ms (7.009us)
- Fired Metastore events: 46.625ms (21.283ms)
Tests:
- Added e2e test
Change-Id: Iaef1cae7e8ca1c350faae8666ab1369717736978
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/query_test/test_observability.py
3 files changed, 13 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/22778/1
--
To view, visit http://gerrit.cloudera.org:8080/22778
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaef1cae7e8ca1c350faae8666ab1369717736978
Gerrit-Change-Number: 22778
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
