Hello Kurt Deschler, Yida Wu, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21541
to look at the new patch set (#7).
Change subject: IMPALA-12906: Incorporate scan range information into the tuple
cache key
......................................................................
IMPALA-12906: Incorporate scan range information into the tuple cache key
This change is accomplishing two things:
1. It incorporates scan range information into the tuple
cache key.
2. It reintroduces deterministic scheduling as an option
for mt_dop and uses it for HdfsScanNodes that feed
into a TupleCacheNode.
The combination of these two things solves several problems:
1. When a table is modified, the list of scan ranges will
change, and this will naturally change the cache keys.
2. When accessing a partitioned table, two queries may have
different predicates on the partition columns. Since the
predicates can be satisfied via partition pruning, they are
not included at runtime. This means that two queries
may have identical compile-time keys with only the scan
ranges being different due to different partition pruning.
3. Each fragment instance processes different scan ranges, so
each will have a unique cache key.
To incorporate scan range information, this introduces a new
per-fragment-instance cache key. At compile time, the planner
now keeps track of which HdfsScanNodes feed into a TupleCacheNode.
This is passed over to the runtime as a list of plan node ids
that contain scan ranges. At runtime, the fragment instance
walks through the list of plan nodes ids and hashes any scan ranges
associated with them. This hash is the per-fragment-instance
cache key. The combination of the compile-time cache key produced
by the planner and the per-fragment-instance cache key is a unique
identifier of the result.
Deterministic scheduling for mt_dop was removed via IMPALA-9655
with the introduction of the shared queue. This revives some of
the pre-IMPALA-9655 scheduling logic as an option. Since the
TupleCacheNode knows which HdfsScanNodes feed into it, the
TupleCacheNode turns on deterministic scheduling for all of those
HdfsScanNodes. Since this only applies to HdfsScanNodes that feed
into a TupleCacheNode, it means that any HdfsScanNode that doesn't
feed into a TupleCacheNode continues using the current algorithm.
The pre-IMPALA-9655 code is modified to make it more deterministic
about how it assigns scan ranges to instances.
Testing:
- Added custom cluster tests for the scan range information
including modifying a table, selecting from a partitioned
table, and verifying that fragment instances have unique
keys
- Added basic frontend test to verify that deterministic scheduling
gets set on the HdfsScanNode that feed into the TupleCacheNode.
- Restored the pre-IMPALA-9655 backend test to cover the LPT code
Change-Id: Ibe298fff0f644ce931a2aa934ebb98f69aab9d34
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/tuple-cache-node.cc
M be/src/exec/tuple-cache-node.h
M be/src/scheduling/scheduler-test.cc
M be/src/scheduling/scheduler.cc
M be/src/scheduling/scheduler.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/common/ThriftSerializationCtx.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java
M tests/custom_cluster/test_tuple_cache.py
15 files changed, 753 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/21541/7
--
To view, visit http://gerrit.cloudera.org:8080/21541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe298fff0f644ce931a2aa934ebb98f69aab9d34
Gerrit-Change-Number: 21541
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Yida Wu <[email protected]>
