[
https://issues.apache.org/jira/browse/IMPALA-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007685#comment-18007685
]
ASF subversion and git services commented on IMPALA-14219:
----------------------------------------------------------
Commit 78a27c56fec29f5f27c24e5b5cd32b454f6dba07 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=78a27c56f ]
IMPALA-13898: Incorporate partition information into tuple cache keys
Currently, the tuple cache keys do not include partition
information in either the planner key or the fragment instance
key. However, the partition actually is important to correctness.
First, there are settings defined on the table and partition that
can impact the results. For example, for processing text files,
the separator, escape character, etc are specified at the table
level. This impacts the rows produced from a given file. There
are other such settings stored at the partition level (e.g.
the JSON binary format).
Second, it is possible to have two partitions pointed at the same
filesystem location. For example,
scale_db.num_partitions_1234_blocks_per_partition_1
is a table that has all partitions pointing to the same
location. In that case, the cache can't tell the partitions
apart based on the files alone. This is an exotic configuration.
Incorporating an identifier of the partition (e.g. the partition
keys/values) allows the cache to tell the difference.
To fix this, we incorporate partition information into the
key. At planning time, when incorporating the scan range information,
we also incorporate information about the associated partitions.
This moves the code to HdfsScanNode and changes it to iterate over
the partitions, hashing both the partition information and the scan
ranges. At runtime, the TupleCacheNode looks up the partition
associated with a scan node and hashes the additional information
on the HdfsPartitionDescriptor.
This includes some test-only changes to make it possible to run the
TestBinaryType::test_json_binary_format test case with tuple caching.
ImpalaTestSuite::_get_table_location() (used by clone_table()) now
detects a fully-qualified table name and extracts the database from it.
It only uses the vector to calculate the database if the table is
not fully qualified. This allows a test to clone a table without
needing to manipulate its vector to match the right database. This
also changes _get_table_location() so that it does not switch into the
database. This required reworking test_scanners_fuzz.py to use absolute
paths for queries. It turns out that some tests in test_scanners_fuzz.py
were running in the wrong database and running against uncorrupted
tables. After this is corrected, some tests can crash Impala. This
xfails those tests until this can be fixed (tracked by IMPALA-14219).
Testing:
- Added a frontend test in TupleCacheTest for a table with
multiple partitions pointed at the same place.
- Added custom cluster tests testing both issues
Change-Id: I3a7109fcf8a30bf915bb566f7d642f8037793a8c
Reviewed-on: http://gerrit.cloudera.org:8080/23074
Reviewed-by: Yida Wu <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> test_scanners_fuzz.py runs some queries against the wrong database with
> uncorrupted tables
> ------------------------------------------------------------------------------------------
>
> Key: IMPALA-14219
> URL: https://issues.apache.org/jira/browse/IMPALA-14219
> Project: IMPALA
> Issue Type: Task
> Components: Test
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> When working on IMPALA-13898, I discovered that
> "ImpalaTestSuite::_get_table_location()" actually switches into a database
> based on the vector (i.e. parquet switches into functional_parquet). When I
> changed this behavior to not switch databases, this broke some tests in
> test_scanners_fuzz.py. It turns out the tests were not running against the
> corrupted version of the tables. When they do run against the corrupted
> version of the table, Impala frequently crashes.
> Basically, the test creates a corrupted table in a unique_database based on
> an original base table and then runs SQLs against it. Some of the SQLs use
> relative names for the corrupted table (e.g. complextypestbl or
> alltypesagg_parquet_v2_uncompressed), which is the same as the base table.
> "ImpalaTestSuite::_get_table_location()" was switching them into
> functional_parquet which has tables of those names, and then nothing else
> ever switched them to the unique_database with the corrupted versions of the
> tables.
> We need to fix this test behavior, but we need to fix the Impala crashes
> before we can make these tests do the right thing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
