Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22215
to look at the new patch set (#3).
Change subject: IMPALA-13478: Sync tuple cache files to disk asynchronously
......................................................................
IMPALA-13478: Sync tuple cache files to disk asynchronously
When a tuple cache entry is first being written, we want to
sync the contents to disk. Currently, that happens on the
fast path and delays the query results, sometimes significantly.
This moves the Sync() call off of the fast path by passing
the work to a thread pool. The threads in the pool open
the file, sync it to disk, then close the file. If anything
goes wrong, the cache entry is evicted.
As basic back pressure, if the thread pool is overwhelmed
and the queue is full, it abandons the write of the cache
entry.
This adds two counters at the daemon level to track the
number of syncs that fail (e.g. due to IO errors) and
the number of syncs that are dropped due to backpressure.
Testing:
- Added a unit test to tuple-cache-mgr-test
- Testing with TPC-DS on a cluster with fast NVME SSDs showed
a significant improvement in the first-run times
Change-Id: I646bb56300656d8b8ac613cb8fe2f85180b386d3
---
M be/src/exec/tuple-file-writer.cc
M be/src/runtime/tuple-cache-mgr-test.cc
M be/src/runtime/tuple-cache-mgr.cc
M be/src/runtime/tuple-cache-mgr.h
4 files changed, 145 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/22215/3
--
To view, visit http://gerrit.cloudera.org:8080/22215
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I646bb56300656d8b8ac613cb8fe2f85180b386d3
Gerrit-Change-Number: 22215
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
