[email protected] has posted comments on this change. ( http://gerrit.cloudera.org:8080/21031 )
Change subject: IMPALA-12709: Add support for hierarchical metastore event processing ...................................................................... Patch Set 29: (7 comments) http://gerrit.cloudera.org:8080/#/c/21031/27//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21031/27//COMMIT_MSG@72 PS27, Line 72: 1. We need to redefine the lag in the hierarchical processing mode. > Please mention FE tests need some changes to run on the new mode. Maybe aft Done http://gerrit.cloudera.org:8080/#/c/21031/20/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/21031/20/be/src/catalog/catalog-server.cc@217 PS20, Line 217: nable_hierarchical_event_proc > The startup flags are used as text, e.g. in scripts or configuration files. Done. Have removed hms_event_polling_interval_ms and changed hms_event_polling_interval_s type to double. http://gerrit.cloudera.org:8080/#/c/21031/27/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/21031/27/be/src/catalog/catalog-server.cc@243 PS27, Line 243: "from the respective executor. Whenever a new event is enqueued to the processor, " > For testing purpose, we might need one more flag to control the max number Can achieve it with db_event_executors and table_event_executors values. Have added tests for rename table now. http://gerrit.cloudera.org:8080/#/c/21031/27/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/21031/27/common/thrift/metrics.json@3616 PS27, Line 3616: "key" : "events-processor.lag-time" > We need to redefine what the lag is in the new mode. Or maybe add more metr Done http://gerrit.cloudera.org:8080/#/c/21031/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/21031/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1275 PS27, Line 1275: List<NotificationEvent> events = getNextMetastoreEvents(currentEventId); > nit: please don't wrap this. I don't think we have existing codes in this s Done http://gerrit.cloudera.org:8080/#/c/21031/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1583 PS27, Line 1583: throw new MetastoreNotificationException(String.format( > Please add TODOs to improve/fix these logs in the new mode, and mention it Done http://gerrit.cloudera.org:8080/#/c/21031/27/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java: http://gerrit.cloudera.org:8080/#/c/21031/27/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@139 PS27, Line 139: EventProcessorStatus.ACTIVE) { > Since TableEventExecutor#process() is scheduled every 10ms, we need to avoi Changed it to trace log -- To view, visit http://gerrit.cloudera.org:8080/21031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 Gerrit-Change-Number: 21031 Gerrit-PatchSet: 29 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Thu, 02 Jan 2025 09:49:10 +0000 Gerrit-HasComments: Yes
