[
https://issues.apache.org/jira/browse/IMPALA-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-13340.
-------------------------------------
Fix Version/s: Impala 4.5.0
Resolution: Fixed
Resolving this. Thank [~MikaelSmith] and [~jasonmfehr] for the review!
> COPY TESTCASE in LocalCatalog mode doesn't dump the partition and file
> metadata
> -------------------------------------------------------------------------------
>
> Key: IMPALA-13340
> URL: https://issues.apache.org/jira/browse/IMPALA-13340
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 4.5.0
>
>
> IMPALA-11901 fixes the failures of using COPY TESTCASE statements in
> LocalCatalog mode. However, only the table metadata is dumped, e.g. table
> schema, column stats. The partition and file metadata are missing.
> To reproduce the issue locally, start the Impala cluster in LocalCatalog mode.
> {code:bash}
> bin/start-impala-cluster.py --catalogd_args=--catalog_topic_mode=minimal
> --impalad_args=--use_local_catalog{code}
> Dump the metadata of a query on a partitioned table:
> {noformat}
> copy testcase to '/tmp' select * from functional_parquet.alltypes;
> +--------------------------------------------------------------------------------------+
> | Test case data output path
> |
> +--------------------------------------------------------------------------------------+
> |
> hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1
> |
> +--------------------------------------------------------------------------------------+
> {noformat}
> Check the metadata from the source cluster
> {noformat}
> show partitions functional_parquet.alltypes
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
> | year | month | #Rows | #Files | Size | Bytes Cached | Cache
> Replication | Format | Incremental stats | Location
> | EC Policy |
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
> | 2009 | 1 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1 |
> NONE |
> | 2009 | 2 | -1 | 1 | 8.09KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2 |
> NONE |
> | 2009 | 3 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3 |
> NONE |
> | 2009 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4 |
> NONE |
> | 2009 | 5 | -1 | 1 | 8.55KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5 |
> NONE |
> | 2009 | 6 | -1 | 1 | 8.23KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6 |
> NONE |
> | 2009 | 7 | -1 | 1 | 8.25KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7 |
> NONE |
> | 2009 | 8 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8 |
> NONE |
> | 2009 | 9 | -1 | 1 | 8.41KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9 |
> NONE |
> | 2009 | 10 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10 |
> NONE |
> | 2009 | 11 | -1 | 1 | 8.44KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11 |
> NONE |
> | 2009 | 12 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12 |
> NONE |
> | 2010 | 1 | -1 | 1 | 8.38KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1 |
> NONE |
> | 2010 | 2 | -1 | 1 | 7.80KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=2 |
> NONE |
> | 2010 | 3 | -1 | 1 | 8.29KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=3 |
> NONE |
> | 2010 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=4 |
> NONE |
> | 2010 | 5 | -1 | 1 | 8.62KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=5 |
> NONE |
> | 2010 | 6 | -1 | 1 | 8.26KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=6 |
> NONE |
> | 2010 | 7 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=7 |
> NONE |
> | 2010 | 8 | -1 | 1 | 8.64KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=8 |
> NONE |
> | 2010 | 9 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=9 |
> NONE |
> | 2010 | 10 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=10 |
> NONE |
> | 2010 | 11 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=11 |
> NONE |
> | 2010 | 12 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
> | PARQUET | false |
> hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=12 |
> NONE |
> | Total | | -1 | 24 | 201.52KB | 0B |
> | | |
> | |
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+{noformat}
> Import the metadata
> {noformat}
> copy testcase from
> 'hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1';
> +----------------------------------------------------------------------------------------------------------------+
> | summary
> |
> +----------------------------------------------------------------------------------------------------------------+
> | Testcase generated using Impala version 4.5.0-SNAPSHOT. 1 db(s), 1 table(s)
> and 0 view(s) imported for query: |
> |
> |
> | SELECT * FROM functional_parquet.alltypes
> |
> +----------------------------------------------------------------------------------------------------------------+{noformat}
> Check the partition metadata, they are missing:
> {noformat}
> set PLANNER_TESTCASE_MODE=true;
> show partitions functional_parquet.alltypes;
> Query: show partitions functional_parquet.alltypes
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
> | year | month | #Rows | #Files | Size | Bytes Cached | Cache Replication |
> Format | Incremental stats | Location | EC Policy |
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
> | Total | | -1 | 0 | 0B | 0B | |
> | | | |
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
