(impala) branch master updated: IMPALA-13453: Avoid reloading partition if it is unchanged

[joemcdonnell]

This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new a7efa7665 IMPALA-13453: Avoid reloading partition if it is unchanged
a7efa7665 is described below

commit a7efa7665fc113ec13a53a77efcc2d9138d44330
Author: Sai Hemanth Gantasala <saihema...@cloudera.com>
AuthorDate: Fri May 30 12:31:02 2025 -0700

    IMPALA-13453: Avoid reloading partition if it is unchanged
    
    In table level REFRESH, we check whether the partition is actually
    changed and skip updating unchanged partitions in catalog. However, in
    partition REFRESH, we always drop and add the partition. This leads to
    unecessarily dropping the partition metadata, column statistics and
    adding them back again. This patch adds a check to verify if the
    partition really changed before reloading the partition to avoid
    unnecessary drop-add sequence.
    
    Change-Id: I72d5d20fa2532d49313d5e88f2d66f98b9537b2e
    Reviewed-on: http://gerrit.cloudera.org:8080/22962
    Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
    Reviewed-by: Quanlong Huang <huangquanl...@gmail.com>
---
 fe/src/main/java/org/apache/impala/catalog/HdfsTable.java | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 
b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
index 7f6cf7f85..77110117b 100644
--- a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
@@ -2847,6 +2847,11 @@ public class HdfsTable extends Table implements 
FeFsTable {
           || HdfsPartition.comparePartitionKeyValues(
           oldPartition.getPartitionValues(), partBuilder.getPartitionValues()) 
== 0);
       if (oldPartition != null) {
+        // check and skip
+        boolean partitionUnChanged = 
partBuilder.equalsToOriginal(oldPartition);
+        LOG.trace("Partition {} {}", oldPartition.getName(),
+            partitionUnChanged ? "unchanged" : "changed");
+        if (partitionUnChanged) continue;
         partBuilder.setPrevId(oldPartition.getId());
         partBuilder.setFileDescriptors(oldPartition);
         partBuilder.setCreateEventId(oldPartition.getCreateEventId());