Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21665 )
Change subject: IMPALA-12865: Fix wrong lastRefreshEventId set by firing RELOAD events ...................................................................... Patch Set 25: (3 comments) http://gerrit.cloudera.org:8080/#/c/21665/20/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21665/20/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7576 PS20, Line 7576: } else { > Ack Thanks for adding the test but I just realized the RELOAD events will be batched and skipped by one self-event check. This seems another issue. Filed IMPALA-14677 for more discussion. http://gerrit.cloudera.org:8080/#/c/21665/22/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21665/22/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7573 PS22, Line 7573: > Ack The new code is slightly different than this. Is there a problem of the above code? Note that on the above last line, we add the in-flight version to the partition (it might be a new instance updated in the if-clause). It seems OK and we don't need to extend the HdfsTable.updatePartition() method. Or did I missing something? http://gerrit.cloudera.org:8080/#/c/21665/22/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21665/22/tests/custom_cluster/test_events_custom_configs.py@1329 PS22, Line 1329: assert self.client.is_finished(handle) > Ack Not done yet. What we need is a test like this: execute_async: refresh tbl partition(year=2024); run_stmt_in_hive: create table tbl2 (i int); verify the RELOAD event is skipped. What are expected to happen: Let's say, before REFRESH, the latest event id is N (currentHMSEventId). REFRESH reloads the table. The CREATE_TABLE event id is N+1. The RELOAD event is fired later due to the sleep. Its event id is N+2. Due to currentHMSEventId + 1 != RELOAD event id, the REFRESH statement just sets lastRefreshEventId to N. When processing the RELOAD event, its event id (N+2) is newer than the lastRefreshEventId. If it still got skipped, we know the in-flight event list checking (i.e. self-event detection) takes effect. -- To view, visit http://gerrit.cloudera.org:8080/21665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90039da77ec561c5aede44456f88c6650582815b Gerrit-Change-Number: 21665 Gerrit-PatchSet: 25 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Pranav Lodha <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Mon, 12 Jan 2026 14:10:40 +0000 Gerrit-HasComments: Yes
