[
https://issues.apache.org/jira/browse/IMPALA-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved IMPALA-13453.
--------------------------------------------
Fix Version/s: Impala 4.5.1
Resolution: Fixed
> REFRESH <table> PARTITION <partition> always update the partition
> -----------------------------------------------------------------
>
> Key: IMPALA-13453
> URL: https://issues.apache.org/jira/browse/IMPALA-13453
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Major
> Fix For: Impala 4.5.1
>
>
> In table level REFRESH, we check whether the partition is actually changed
> and skip updating unchanged partitions in catalog:
> [https://github.com/apache/impala/blob/42fda24364786cc1a457890bd212bb3922479e95/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1098-L1101]
> {code:java}
> public void updatePartition(HdfsPartition.Builder partBuilder) throws
> CatalogException {
> HdfsPartition oldPartition = partBuilder.getOldInstance();
> ...
> boolean partitionNotChanged = partBuilder.equalsToOriginal(oldPartition);
> LOG.trace("Partition {} {}", oldPartition.getName(),
> partitionNotChanged ? "changed" : "unchanged");
> if (partitionNotChanged) return;
> HdfsPartition newPartition = partBuilder.build();
> // Partition is reloaded and hence cache directives are not dropped.
> dropPartition(oldPartition, false);
> addPartition(newPartition);
> }{code}
> However, in partition REFRESH, we always drop and add the partition:
> [https://github.com/apache/impala/blob/42fda24364786cc1a457890bd212bb3922479e95/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L3093-L3096]
> {code:java}
> for (Map.Entry<HdfsPartition.Builder, HdfsPartition> entry :
> partBuilderToPartitions.entrySet()) {
> if (entry.getValue() != null) {
> dropPartition(entry.getValue(), false);
> }
> addPartition(entry.getKey().build());
> }{code}
> We should add the same check to avoid updating unchanged partitions.
> CC [~csringhofer], [~hemanth619]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
