This is an automated email from the ASF dual-hosted git repository. stigahuang pushed a commit to branch branch-4.1.1 in repository https://gitbox.apache.org/repos/asf/impala.git
commit b92a8ad745e2499103a55aeb5e58aa090aee05e6 Author: stiga-huang <[email protected]> AuthorDate: Tue Sep 20 10:41:24 2022 +0800 IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables When applying ALTER_PARTITION events on transactional tables, we refresh the partition using the metadata in events if hms_event_incremental_refresh_transactional_table is enabled (which is the default). This could be wrong if the ALTER_PARTITION event is stale. The partition metadata will be rolled back to a stale state. This patch compares the eventId with the createEventId of the table and ignores those ALTER_PARTITION events that have older (smaller) event ids. Note that we already do this for many other event types, ALTER_PARTITION is somehow missing the checks. Eventually we should depend on the lastSyncedEventId and replace createEventId with it. The self-event detection can also be replaced since self-events are also stale events. These will be addressed in IMPALA-10976. Tests - Verified locally with local-catalog mode and event-processor enabled and iterated test_acid_compute_stats for 1400 times. Without the fix, the test would fail in tens of runs. Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Reviewed-on: http://gerrit.cloudera.org:8080/19020 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/19126 Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Quanlong Huang <[email protected]> --- fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java index 43c9a50a1..d3b0e8be6 100644 --- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java +++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java @@ -4368,6 +4368,11 @@ public class CatalogOpExecutor { if (!(table instanceof HdfsTable)) { throw new CatalogException("Partition event received on a non-hdfs table"); } + if (eventId > 0 && eventId <= table.getCreateEventId()) { + LOG.debug("Not reloading partitions of table {}.{} for event {} since it is " + + "recreated at event {}.", dbName, tblName, eventId, table.getCreateEventId()); + return 0; + } try { tryWriteLock(table, reason); long newCatalogVersion = catalog_.incrementAndGetCatalogVersion();
