Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21665 )
Change subject: IMPALA-12865: Fix wrong lastRefreshEventId set by firing RELOAD events ...................................................................... Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/21665/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21665/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7174 PS16, Line 7174: currentHMSEventId = eventIds.get(0); > I want to update currentHMSEventId, if the latestEventId and currentEventId Is this required to fix the test failure? It seems we can further optimize it that if there are no events on this table between currentHMSEventId and eventIds.get(0), we can set lastRefreshEventId to eventIds.get(0) to skip processing this RELOAD event. However, it's hard to check this for transactional tables since CommitTxnEvent and AbortTxnEvent don't have the table names. Maybe we can just do it for non-transactional tables. We can discuss this in a future JIRA. More importantly, we need to improve the logic of skipping self RELOAD events. Currently we just check the lastRefreshEventId: https://github.com/apache/impala/blob/d086babdbd249df0069900739f24da280b06a279/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L3134-L3137 This is sufficient to skip self-events before we fix the wrong lastRefreshEventId. Now after this fix, we can only skip the ReloadEvent when it's exactly the next event after currentHMSEventId. This might be rare in a real cluster that have lots of activities. Then each REFRESH will reload the table twice. The second reload happens in processing the self RELOAD event. I think we should avoid this regression in this patch. Maybe reusing the InFlightEvents used in other event types. I would like to hear your thoughts. -- To view, visit http://gerrit.cloudera.org:8080/21665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90039da77ec561c5aede44456f88c6650582815b Gerrit-Change-Number: 21665 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Thu, 12 Dec 2024 13:03:55 +0000 Gerrit-HasComments: Yes
